Does that SEO agency block AI crawlers? Check in two minutes

Why this test is worth two minutes

Nearly every agency pitching Southwest Florida businesses now lists some version of AI search optimization: getting your roofing company mentioned in ChatGPT, your dental practice cited by Perplexity, your dock-building outfit surfaced in Google's AI Overviews. Claims like that are cheap to make. A robots.txt file is a receipt, and it's public.

robots.txt is a plain text file that sits at the root of every website and tells crawlers which parts of the site they may fetch. It is public by design; you can read anyone's, no login, no tools. If an agency's own robots.txt turns away the AI crawlers, one of two things is true. Either they don't know (a plugin or security setting blocked the bots and nobody noticed), or they know and haven't fixed it. Neither is a great trait in the people about to configure crawler access on your site.

Full disclosure before we go further: this page lives on an SEO agency's site, and AI search work is a core part of what we sell. So judge us by the same test. Our own robots.txt is covered near the end of this page, and we'd genuinely rather you check it than take our word.

How to run the test, step by step

1. Get the exact domain. Pull it from the agency's proposal, email signature, or Google Business Profile listing.
2. Load the file. In your browser's address bar, type the domain followed by /robots.txt and press enter. Example: example-agency.com/robots.txt.
3. Don't be thrown by the plain text. You'll see unstyled lines starting with User-agent:, Allow:, and Disallow:. That's what the file is supposed to look like.
4. Search for the bot names. Press Ctrl+F (Cmd+F on a Mac) and check for each of these, one at a time: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot.
5. Read the line underneath each hit. Disallow: / means that bot is barred from the entire site. Disallow: with nothing after it, or Allow: /, means the bot is welcome.
6. Check the wildcard too. Search for User-agent: *. If that block ends in Disallow: /, the site is turning away every crawler, AI or otherwise, that obeys the file.

Here's what a block looks like in the wild, so you know it when you see it:

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Two lines per bot. That's the whole tell. A site carrying those lines has asked OpenAI's and Perplexity's crawlers to stay out entirely.

Who these bots are and why each one matters

• GPTBot is OpenAI's main crawler. Blocking it keeps a site's content out of the systems behind ChatGPT.
• OAI-SearchBot and ChatGPT-User are the fetchers OpenAI uses for search results and live browsing. Blocking these cuts a site out of ChatGPT's web-backed answers specifically, which is exactly where "get recommended by ChatGPT" pitches live.
• ClaudeBot is Anthropic's crawler, which feeds Claude.
• PerplexityBot belongs to Perplexity, which retrieves pages live and shows numbered citations. A site it can't fetch is a site it can't cite.
• Google-Extended is Google's opt-out token for AI training. As of mid-2026, blocking it does not remove a site from regular Google search, and AI Overviews draw on the normal Google index either way. So a Google-Extended block is less damaging than the others, but on an agency's own site it still tells you where they stand.
• CCBot is Common Crawl, an open dataset of the web that many AI systems learn from.

One more nuance worth knowing: as of mid-2026, ChatGPT's search feature leans on Bing's index, so a site that blocked bingbot would hurt its AI visibility from a second direction. That block is rarer, but it takes one extra Ctrl+F to rule out while you're in the file.

The blocks robots.txt won't show you

Here's the catch that makes this a two-part test instead of a one-part test: robots.txt is a request, not a wall. A site can also block crawlers at the server or firewall level, and that kind of block leaves no trace in the file you just read.

The most common source is Cloudflare, which sits in front of a large share of business websites and offers a setting that blocks known AI crawlers at the network edge. It shows up two ways:

• Managed robots.txt lines. Cloudflare can append AI-bot block directives to a site's robots.txt automatically, usually with a comment noting they're managed. If you see a tidy block of AI bot names with an auto-generated note above it, someone flipped a toggle. Maybe deliberately, maybe not, and that distinction is worth asking about.
• Edge blocks behind a clean robots.txt. The site can return a 403 Forbidden to AI crawlers while the robots.txt reads as wide open. The file says welcome; the firewall says no.

The 30-second follow-up for hidden blocks

If you're comfortable opening a terminal, ask the site for its homepage while identifying as an AI crawler:

curl -A "GPTBot" -I https://example-agency.com/

Look at the first line of the response. 200 means the server handed over the page. 403 means it refused, and if a normal browser loads the same page fine, you've found a server-level AI block that robots.txt never mentioned.

If a terminal isn't your thing, our free AI Visibility Checker runs this kind of access check automatically, and it works on an agency's homepage just as well as it works on yours.

How to read what you find

A clean robots.txt doesn't prove an agency is good at AI search. It's a floor, not a ceiling, and plenty of sites pass by default because nobody ever touched the file. Passing this test simply earns an agency the right to take the harder one: showing you dated, reproducible examples of AI engines citing their work.

Failing is more informative, and not every failure means the same thing:

• Training bots blocked, retrieval bots allowed. Blocking GPTBot and CCBot while allowing OAI-SearchBot and PerplexityBot is a coherent editorial stance: no training on our content, but answer engines may cite us. It's an odd posture for a firm selling AI visibility, but it shows they understand the layers. Ask them to walk you through the choice; a crisp answer is a good sign.
• Retrieval and search bots blocked. Blocking OAI-SearchBot, ChatGPT-User, or PerplexityBot while pitching AI search optimization is the contradiction this test exists to catch. The agency is invisible to the exact answer engines it's promising to put you in.
• Everything blocked, AI line item on the proposal. Walk into the sales call with questions, not a signature.

This isn't hypothetical in our market. We documented a case of it here in Southwest Florida: a documented local example of a firm marketing AI services while its own site turned the crawlers away.

What to say in the sales meeting

You don't need to be confrontational about it. One sentence does the work:

"I looked at your robots.txt before this call, and it disallows GPTBot and PerplexityBot. Walk me through that choice."

Good answers exist. "We block training bots on purpose, and here's why" is one. "Good catch, that's a Cloudflare default we should have caught, here's the fix" is honest, if less flattering. Bad answers include not knowing what robots.txt is, blaming an unnamed plugin without offering a fix, or insisting crawler access doesn't matter for the AI results they're selling.

Fold the question into the rest of your vetting. We keep a full list of questions to ask before hiring an SEO company, and this check is step one in our larger guide to choosing an SEO company for a reason: it's the only vetting step that takes two minutes and can't be spun.

Our own robots.txt, since you're about to check

Fair's fair. naplesseo.com welcomes GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, and Google-Extended by name in robots.txt, and we publish llms.txt and llms-full.txt files for AI systems on top of that (verified July 2026). AI search optimization is included in every plan we sell, so being fully crawlable ourselves is table stakes, not a flex. Run the test on us first if you like; it's good practice for running it on everyone else.

Then run the same test on your own site

The same two minutes will tell you whether your own roofing, plumbing, or dental site is turning AI engines away right now, often thanks to a security plugin or a Cloudflare toggle nobody remembers flipping. The full walkthrough for your own site lives at check if AI can read your site, and if you find blocks, here's how to remove them.

A Naples pool builder or a Marco Island dock contractor deciding between agencies this month has plenty of hard judgment calls ahead. This one isn't one of them. Two minutes, one text file, and you'll know whether the AI pitch in front of you is practiced or just printed.

Frequently asked questions