robots.txt

robots.txt tells search engine crawlers which pages they can and cannot access.

In this section

• What is robots.txt
• Check details
• Common issues
• Platform guides: WordPress · Shopify · Joomla

Quick facts

• Location: https://yoursite.com/robots.txt
• Format: Plain text with User-agent, Allow, Disallow, and Sitemap directives
• Specification: robotstxt.org