robots.txt check details¶
Checks performed¶
| Check | Weight | Pass | Warn | Fail |
|---|---|---|---|---|
| File exists and is reachable | 25 | HTTP 200 | Redirect | 404 or unreachable |
| Valid syntax | 20 | All directives parse | Minor warnings | Unparseable |
| Sitemap directive present | 15 | At least one Sitemap: line |
— | No sitemap declared |
| No blanket disallow | 15 | Not blocking * |
— | All crawlers blocked |
| Consistent protocol | 10 | All HTTPS | Mixed | — |
| Reasonable file size | 5 | Under 500KB | 250–500KB | Over 500KB |
| Valid sitemap URLs | 10 | All return 200 | — | Errors |