CiteStat Bot (our crawler)
TL;DR
CiteStat's crawler identifies itself as ERPFitSEOBot / CiteStat, follows your robots.txt when it crawls more than the page you asked for, and only runs when someone requests a scan.
Explain Simply
Our bot is a visitor who knocks first. Before it walks into other rooms of your site, it reads the 'Staff Only' signs in robots.txt and stays out where they say so.
CiteStat fetches pages only when a user asks for a scan of a URL. A single-page scan fetches that one URL, plus the standard root files (robots.txt, sitemap.xml, llms.txt) it audits.
A multi-page scan (quick, sample or full mode) discovers more URLs from the sitemap and homepage links. Before it fetches any of them, it reads robots.txt and skips every URL that robots.txt disallows for our crawler. The report lists those skipped URLs as "blocked by robots.txt", because a page you block cannot be indexed either.
User-agent tokens: page fetches send "AiSEO/2.0", status checks send "ERPFitSEOBot/1.0"; both carry a link to this page. A robots.txt group for either token applies to us; otherwise we follow the "User-agent: *" group. Crawls are small (10–50 pages) and run with low concurrency.
Bot simulation is separate: to show what Googlebot or Bingbot would receive, CiteStat fetches the ONE URL a user submits with that search crawler's user-agent string. It is a simulation by CiteStat — we are not Google or Microsoft — and those user-agents are never used in multi-page crawls.
⚡Why It Matters
A crawler that ignores robots.txt wastes your server's resources and reads pages you chose to keep out of search. Honouring robots.txt is the basic contract between a site and any bot — and it lets the report show you exactly which URLs your own rules hide from search engines.
🔍How CiteStat Checks
To block CiteStat's multi-page crawl, add a group to robots.txt:
User-agent: ERPFitSEOBot Disallow: /
User-agent: CiteStat Disallow: /
A path rule such as "Disallow: /admin/" keeps us out of that section only. The URL a user enters directly is still fetched once, like a visit in a browser.
CiteStat Score Impact
Not scored. URLs blocked by robots.txt appear as an info-level site-wide issue in multi-page scans.
Fix Difficulty
Editor: N/A | Developer: Easy | AI Agent: Easy