Skip to content
FixAEO
Free tool · No signup · Runs in your browser

Free Robots.txt File Checker

Paste your robots.txt. See which AI + search crawlers you're allowing or blocking, with concrete AEO recommendations.

Parser and crawler list reviewed August 1, 2026 · your file stays in this browser

Paste your robots.txt

Drop the file content

Open yoursite.com/robots.txt in your browser, copy everything, paste here.

Need a starter robots.txt? Use our Robots.txt Generator.

Paste your robots.txt content on the left. We'll parse it, check each AI + search bot's status, and flag AEO issues.

What this robots.txt validator checks

Three things, in plain English — per-bot verdicts, AEO red flags, and syntax errors. It summarizes crawler access across the file; it does not simulate the longest-rule match for a specific page URL.

19 known crawlers

Major AI search bots, AI training bots, and classic search engines — summarized against their named rules and the catch-all.

AEO red flags for AI bots

Blocking the index-time AI bots, blocking Googlebot, missing Sitemap directive, malformed lines — all surfaced as warnings or errors.

One-click report

Copy a clean markdown report of every bot verdict + issues. Paste straight into a Notion doc or ticket.

How the robots.txt test works

  1. The parser separates user-agent groups and reads Allow, Disallow and Sitemap directives.
  2. Each of the 19 listed crawlers is checked against its named group plus the * fallback.
  3. The tool flags blocked citation crawlers, a blocked Googlebot, missing sitemaps and malformed directives.
  4. Results are calculated locally. Nothing is submitted to FixAEO.

Because this is a pasted-file audit, use Google Search Console's live URL tools or the crawler provider's tester when you need a definitive verdict for one exact URL.

Frequently asked questions

Why paste instead of fetch by URL?
Browser CORS policy blocks JavaScript on this page from fetching robots.txt on third-party domains. So you paste it manually — open yoursite.com/robots.txt, copy the whole file, paste here. Nothing leaves your browser. The full FixAEO audit (server-side) does fetch robots.txt automatically for you.
What does 'Partial' verdict mean for a bot?
The bot is allowed in general, but some paths are blocked for it (or for *, which applies to it). It's not fully blocked — but specific URLs may be off-limits. Click into your file to see which Disallow paths are in effect.
Which AI bots matter most?
For AEO: OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot. These are the index-time bots that power live citations. Training bots (GPTBot, Google-Extended, Bytespider, CCBot) are a separate decision — blocking them opts you out of model training without affecting today's citations.
Will blocking AI bots hurt SEO?
Classic SEO no — Googlebot is a separate user-agent from Google-Extended. You can block AI training and keep all your Google Search rankings. AEO yes — blocking the index-time AI bots removes you from AI assistant citations entirely.
Why isn't my custom bot listed?
We report on 19 common AI and search crawlers. Rules for other user-agents remain visible in the parsed file, but they do not receive a per-bot summary card. This checker gives a site-wide access summary; test a specific URL with the crawler's own live testing tool when path-level confirmation matters.

Need a starter file?

Our Robots.txt Generator builds an AEO-friendly starter file in 30 seconds, with separate presets for search, citation and training crawlers.

Open Robots.txt Generator
Was this helpful?