Common robots.txt issues¶

Blanket disallow¶

Disallow: / blocks all crawlers from your entire site — usually left over from staging.

Fix: Change to Allow: / or remove the line. Add specific disallow rules only for pages you want hidden.

Without a Sitemap: line, search engines must discover your sitemap on their own.

Fix: Add Sitemap: https://yoursite.com/sitemap.xml at the bottom.

References to both HTTP and HTTPS URLs confuse crawlers.

Fix: Ensure all URLs use https://.

A 404 means crawlers assume they can access everything — you lose control over crawler behavior.

Fix: Create a robots.txt file at your site root.