Robots.txt
TL;DR
Robots.txt tells search engine crawlers which parts of your site to crawl and which to skip. One wrong line can hide your entire site from Google.
Explain Simply
Robots.txt is like a 'Staff Only' sign at a store. It tells Google's robots which aisles they can browse and which are off-limits. But it's just a sign — sneaky robots can ignore it.
Robots.txt is a text file at the domain root that instructs web crawlers which pages to crawl and which to skip. Uses User-agent, Allow, Disallow directives. Not a security mechanism — just guidance.
The file must be at the exact path /robots.txt. It supports wildcards (*) in paths and the $ end-of-URL anchor. Multiple User-agent blocks can target specific crawlers (Googlebot, Bingbot, GPTBot). The Sitemap directive points crawlers to your XML sitemap.
Common mistakes: blocking CSS/JS files (prevents rendering), using Disallow: / on production (blocks everything), or forgetting to update after migration. Always test changes with Google's robots.txt tester.
⚡Why It Matters
Incorrect robots.txt can block Google from crawling your entire site, or expose sensitive URLs. It also points to sitemap.xml. A misconfigured robots.txt is one of the fastest ways to lose all organic traffic.
🔍How AiSEO Checks
AiSEO fetches and parses robots.txt, checks for Googlebot blocking, verifies sitemap references, and detects common syntax errors.
AiSEO Score Impact
Up to 3 points in the Common Files category (10% of total score)
Fix Difficulty
Editor: N/A | Developer: Easy | AI Agent: Easy