Robots.txt and Sitemaps — Clean, Simple Setup
Updated 10 Aug 2025
Your robots.txt tells crawlers where not to go. Your sitemap.xml lists the URLs that matter. Keep both simple, truthful and up to date.
Minimal robots.txt
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Common mistakes
- Blocking CSS/JS — prevents Google from rendering your pages properly.
- Overusing Disallow — only block what truly must be hidden.
- Wrong sitemap path — ensure the URL is absolute and publicly reachable.
Sitemap tips
- Only indexable pages should be in the sitemap.
- Include <lastmod> where possible.
- If you have multiple sitemaps, use a sitemapindex.
FAQs
Do I need a robots.txt?
It’s optional but recommended. At minimum, allow everything and point to your sitemap.
Should I block /admin/?
Blocking admin paths is fine, but never put sensitive data anywhere public. Use authentication.
Can I list multiple sitemaps?
Yes, either multiple Sitemap lines or a sitemap index that references them all.