Can AI engines read
your website?
The most common reason a brand is invisible in ChatGPT and Perplexity is not bad content. It is a blocked robots.txt. Check five AI crawlers in 30 seconds, get a precise access report and fix code instantly.
checked
check time
parser
We fetch your public robots.txt using a CORS proxy. No data is stored. No login required.
Bot access is the
first audit check.
In every AEO audit I run, the first check is always robots.txt. Not schema. Not content. Not entity markup. Bot access is the hard prerequisite for everything in AI search visibility.
AI engines cannot cite what they cannot read. A brand with perfect schema, answer-first content, and strong topical authority is still invisible in ChatGPT if GPTBot cannot crawl the site. The block is a hard stop before every other optimization.
The irony: fixing bot access takes under 10 minutes. Most brands spend months refining content strategy while a single robots.txt rule sits quietly blocking every AI crawler that matters for citations.
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Bingbot Allow: / User-agent: Googlebot Allow: /
- GPTBot cannot index your content for ChatGPT
- ClaudeBot skips your pages in citation retrieval
- PerplexityBot cannot read your answers
- Bingbot block means Copilot cannot cite you
- Brand invisible in all AI-generated answers
- Competitors with open access get cited instead
- Content investment delivers zero AI citation ROI
- All five AI crawlers can index your content
- Your answers are eligible for ChatGPT citations
- Perplexity can pull your content for buyer queries
- Copilot can surface your brand via Bing index
- Google AI Overviews can include your pages
- Schema and content work compounds as intended
Five crawlers.
Five citation surfaces.
Blocking one bot means missing one AI engine entirely. Most brands block at least two. Here is what each crawler indexes and why each access status matters for your brand's AI visibility.
OpenAI's web crawler for ChatGPT's citation and browsing features. Blocking it means your brand cannot appear when users ask ChatGPT about vendors, tools, or topics in your category. ChatGPT has 200M+ weekly active users, many running vendor research queries.
User-agent: GPTBot Anthropic's crawler for Claude's web search capability. Often inadvertently blocked alongside GPTBot on conservative configurations because both appeared around the same time. Claude is growing as a research tool for B2B buyers.
User-agent: ClaudeBot Perplexity's crawler indexes content for its citation-heavy answer engine. Perplexity is the most citation-aggressive of the major AI search engines: it surfaces multiple sources per answer and drives real referral traffic to cited pages.
User-agent: PerplexityBot Microsoft's crawler powers both Bing search and all Copilot responses across Microsoft 365, Edge, and Windows. A site not indexed by Bing cannot appear in any Copilot response, regardless of content quality or schema. Bing indexation is a separate check from Google.
User-agent: Bingbot Google's crawler indexes for both traditional search and Google AI Overviews. AI Overviews now trigger on approximately 47% of informational queries. Blocking Googlebot removes your brand from both surfaces simultaneously: traditional rankings and AI-generated answers.
User-agent: Googlebot
Which platforms block
AI bots by default.
What I find when I run this check on sites built on popular platforms. Not a platform bug: a configuration gap that defaults to blocking unless you explicitly allow each bot.
New Webflow projects may restrict unlisted user-agents by default. AI crawlers added after 2022 frequently fall into blocked categories unless explicitly allowed in the project's robots.txt settings.
WordPress itself is permissive. Plugins like Yoast SEO generate the robots.txt file and may not include allowances for newer AI crawlers not yet added to their allow-lists.
Shopify auto-generates robots.txt and restricts certain paths by default. AI bots may be grouped alongside restricted crawler categories depending on the store configuration.
No default restrictions. The developer writes the file from scratch. Most Astro builds I ship explicitly allow all five AI crawlers from launch day, making this the cleanest platform for AI visibility.
Bot access is step one.
Here is what comes next.
Fixing robots.txt removes the hard block. What follows is the AEO system that turns bot access into actual citations.
This tool. Allow GPTBot, ClaudeBot, PerplexityBot, Bingbot, and Googlebot explicitly in robots.txt. Then verify Bing Webmaster Tools indexation separately: Copilot relies on Bing, and Bing indexation is independent of Google.
JSON-LD Person and Organization schema establishes your brand identity in AI engine knowledge graphs. Without it, AI engines know your site exists but cannot confidently identify who you are, what you do, or why you are authoritative enough to cite.
Schema for AI search →AI engines cite pages that lead with the direct answer in the first two sentences. If your content buries the answer after context-setting paragraphs, bots read it and still choose a more directly structured competitor page.
How to get cited by ChatGPT →Search 20 of your most important buyer queries in ChatGPT, Perplexity, Claude, and Google AI Overviews monthly. Record which platforms cite you, for which queries, and what the cited content looks like. That signal tells you exactly what to optimize next.
Full AEO strategy →
What I find when
I run this check.
The most consistently underestimated block. Brands check Googlebot first and assume everything else is fine. Copilot runs entirely on Bing. If Bingbot is blocked or the site is not in the Bing index at all, Copilot cannot cite you regardless of what your robots.txt says about GPTBot. I check Bing Webmaster Tools indexation separately on every audit before touching content.
I regularly find robots.txt files with a last-modified date from 2020 or 2021. Disallow: / applies to all user-agents not explicitly listed. Every AI crawler added after that file was written gets blocked by default. GPTBot, ClaudeBot, and PerplexityBot all launched in 2022 or 2023 and are invisible to those old files.
Fixing robots.txt at the CMS or platform level does not always propagate immediately if a CDN or reverse proxy is caching the file. The crawler fetches the cached blocked version and records the block. After any robots.txt fix, check the live response headers for cache-control directives and confirm the updated file is what crawlers actually receive.
Found bots blocked? Build the fix in seconds with the AI Robots.txt Generator. Or pair this checker with the AI Visibility Score quiz for a complete surface check: bot access plus schema, content structure, entity consistency, and Core Web Vitals.
What people ask about
AI bot access.
How the check works, what to do with blocked bots, and how access connects to AI citations.
01Which AI bots does this tool check?
Five: GPTBot (ChatGPT / OpenAI), ClaudeBot (Anthropic / Claude), PerplexityBot (Perplexity AI), Bingbot (Bing / Microsoft Copilot), and Googlebot (Google / AI Overviews). These five crawlers power every major AI engine that cites web sources in responses. A block on any one of them removes you from that engine's citation pool entirely.
02Is this tool safe to run on any domain?
Yes. The checker fetches only the public /robots.txt file using a CORS proxy. No login required, no scraping behind authentication, and no data stored beyond your current session. It reads exactly what any AI crawler would already see when visiting your domain.
03Why are AI bots blocked so often by accident?
Most platforms shipped their default robots.txt configurations before GPTBot, ClaudeBot, and PerplexityBot existed in 2022 and 2023. Conservative rules that block unlisted user-agents or apply Disallow: / to wildcard groups catch these newer crawlers automatically. The site owner rarely knows until they run a check. The fix is three to five lines of robots.txt config.
04What should I do if my bots are blocked?
The tool generates the exact robots.txt additions you need. Copy the lines, add them to your robots.txt file, and redeploy or clear your CDN cache. Most platforms pick up the change within 24 to 48 hours. After fixing access, the next priority is entity schema and answer-first content structure: bot access is the prerequisite, not the full solution.
05Does allowing AI bots affect my Google SEO?
No. GPTBot, ClaudeBot, and PerplexityBot are entirely separate crawlers from Googlebot. Allowing them has zero effect on how Google crawls, indexes, or ranks your content. The only effect is making your content accessible to AI engines that could cite your brand in their responses.
06Is fixing robots.txt enough to get cited in AI search?
No. Bot access is the prerequisite, not the solution. A blocked bot is a hard stop: it cannot cite what it cannot read. But an allowed bot still needs entity schema, answer-first content, and consistent topical signals before it will cite your brand over a competitor. Bot access first, then the full AEO system.
Bot access is the start.
The full system goes deeper.
The free check covers access. A full engagement maps entity schema gaps, content structure problems, and citation tracking across all six AI engines. Replied within 4 hours.