Robots.txt
Robots.txt is a text file at the domain root that instructs web crawlers which pages to crawl and which to skip. Uses User-agent, Allow, Disallow directives. Not a security mechanism — just guidance.
⚡Why It Matters
Incorrect robots.txt can block Google from crawling your entire site, or expose sensitive URLs. It also points to sitemap.xml.
🔍How AiSEO Checks
AiSEO fetches and parses robots.txt, checks for Googlebot blocking, verifies sitemap references, and detects common syntax errors.