AI Crawler Access Checker
Evaluate the robots.txt rules for GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers. Network-level WAF and CDN access is reported separately where it can be observed.
No signup required • 100% free • Robots.txt rules for 20 AI crawlers
If AI crawlers cannot read you, they cannot cite you.
ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews can only recommend pages they can discover and retrieve. If your site blocks important llm crawler traffic, your product pages, guides, pricing pages, and articles can be invisible in AI answers no matter how strong the content is.
Most sites block 2 to 4 major AI crawlers without realizing it. The usual causes are copied robots.txt templates, platform defaults, SEO plugin settings, or broad wildcard rules that catch AI crawlers by accident.
This ai crawlability test helps you decide where to allow AI crawlers and where to block AI crawlers using the robots.txt file crawlers are expected to read first.
Read robots.txt
We parse every user-agent group, including wildcard rules, then test the exact path you entered.
Evaluate each AI bot
We apply robots.txt matching rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and every bot in the list.
Show the fix
You get the blocking rule and copy-paste robots.txt guidance for bots you want to allow.
Not every AI crawler is asking for the same thing.
Training crawlers
Bots like GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, and Meta-ExternalAgent are usually tied to training or improving AI systems. Some site owners allow them because they want broader AI visibility. Others block them because they do not want their content used this way.
If you are deciding what to allow, start by separating these from the bots that fetch pages for a specific user or search answer.
Search and user crawlers
Bots like ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, GoogleOther, Bingbot, MicrosoftPreview, MistralAI-User, DuckAssistBot, and Amazonbot are more closely tied to search, answers, previews, or user-requested browsing.
Many teams choose to allow these so ChatGPT, Claude, Perplexity, Bing, and Google can cite current site content, even if they block separate training crawlers.
Crawlers checked by this tool.
GPTBot
Training crawlerChatGPT-User
Live retrievalOAI-SearchBot
Search crawlerClaudeBot
Training crawlerClaude-User
Live retrievalClaude-SearchBot
Search crawlerPerplexityBot
Search crawlerPerplexity-User
Live retrievalGooglebot
Reference crawlerGoogle-Extended
Training crawlerGoogleOther
Search crawlerBingbot
Reference crawlerMicrosoftPreview
Search crawlerCCBot
Training crawlerApplebot-Extended
Training crawlerMeta-ExternalAgent
Training crawlerAmazonbot
Search crawlerBytespider
Training crawlerMistralAI-User
Live retrievalDuckAssistBot
Search crawlerAI crawler access questions.
How do I check if GPTBot can crawl my website?
Enter your domain in this AI crawler access checker and run the test. We fetch your robots.txt and evaluate GPTBot rules against the exact path. That shows whether GPTBot is allowed or blocked by the robots.txt policy crawlers are expected to follow.
How do I unblock GPTBot or ClaudeBot in robots.txt?
Find the User-agent rule that blocks GPTBot, ClaudeBot, or a wildcard User-agent: * group. Remove the Disallow line or add a more specific Allow line for the paths you want AI crawlers to read. After editing robots.txt, rerun this robots.txt AI checker to confirm the rule changed.
Why does my site block AI crawlers even though robots.txt allows them?
This tool evaluates the public robots.txt policy. Cloudflare, Sucuri, a WAF, a CDN bot rule, or a security plugin can still block requests by IP, geography, session, or crawler verification. An allowed result here is not proof that every network layer will allow the real crawler.
Should I block or allow AI crawlers?
It depends on your goals. If you want AI search visibility, GEO, AEO, citations, and recommendations from ChatGPT, Claude, Perplexity, and Gemini, allow search and retrieval crawlers. If you do not want content used for model training, you may choose to block training crawlers while allowing live answer crawlers.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI crawler traffic used to discover and learn from web content. ChatGPT-User is tied to user-requested browsing or retrieval inside ChatGPT. Many sites allow ChatGPT-User for live answers while making a separate decision about GPTBot and long-term training access.
Which AI crawlers should I allow for ChatGPT, Claude, Perplexity, and Gemini visibility?
For ChatGPT search visibility, prioritize OAI-SearchBot; GPTBot controls potential training use, while ChatGPT-User represents user-requested retrieval. For Claude, review ClaudeBot, Claude-User, and Claude-SearchBot. For Perplexity, review PerplexityBot; Perplexity says its user-triggered fetcher generally ignores robots.txt. Googlebot controls Google Search and its AI search features, while Google-Extended applies to some other Gemini and generative AI uses.
Does blocking Google-Extended affect my Google rankings?
No. Google says Google-Extended is separate from Google Search indexing. Googlebot, not Google-Extended, controls eligibility for Google Search features including AI Overviews and AI Mode. Google-Extended applies to training and grounding in some other Gemini and generative AI systems.
What is the difference between robots.txt and llms.txt?
Robots.txt is an established crawler access policy. llms.txt is an emerging Markdown proposal for publishing a site summary and curated links; it does not grant access, and support varies by AI product.
How often should I check AI crawler access?
Check after every robots.txt change, CDN or WAF change, platform migration, security plugin update, or theme deployment. For ecommerce and content sites, a monthly AI crawlability test is reasonable because Cloudflare, WordPress, Shopify, and CDN defaults can change without obvious page-level symptoms.
Does Cloudflare block AI bots by default?
Cloudflare gives site owners controls for blocking AI bots, and many sites enable those controls directly or inherit them through managed bot settings. Those blocks are not always visible to an outside checker. Review Cloudflare settings if you need to confirm enforcement beyond what robots.txt says.
Inspect any website.
Find sitemaps, validate them, discover llms.txt files, and test AI crawler access.
Free. No signup.