The AI Search Directory
A reference catalogue · Vol. I
Free AI search tool · Crawler access check

AI Crawler Access Checker

Enter a domain. We fetch the live robots.txt, run it against the 12 crawlers that feed AI answers today, and tell you what's open, what's blocked, and what to fix first.

robots.txt · 12 AI crawlers · ~2s server fetch
GPTBotClaudeBotPerplexityBotGoogle-ExtendedOAI-SearchBot+ 7 more
Enter a domain above to run the first check.

About this instrument.

i.

What it actually tests.

We fetch your live robots.txt server-side, then evaluate it against the user-agent strings of the crawlers that feed today's AI answer engines - including GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and OAI-SearchBot.

ii.

Why it's the first step.

Visibility tools optimise content and track citations. None of that helps if the crawler can't reach the page. A blocked Disallow is the cheapest mistake to find and the most expensive one to ignore.

iii.

What you can do with it.

Share the report with a developer or SEO lead before changing crawl policy. Re-run after edits to confirm the fix. Use it as a baseline check before investing in monitoring or citation tooling.