Can AI actually read your website? Five checks that take fifteen minutes

Why this is the first question, not the tenth

Every AI visibility tactic you've heard about, from schema to answer-shaped content to Google Business Profile cleanup, sits downstream of one boring requirement: the engines have to be able to fetch your pages. Optimizing a site AI can't read is painting a locked showroom. The work might be gorgeous. Nobody making a buying decision will ever step inside to see it.

As of mid-2026, the systems that recommend local businesses all do their reading slightly differently. ChatGPT search leans on Bing's index plus OpenAI's own retrieval crawler. Perplexity fetches pages live and cites them as numbered sources. Google's AI Overviews draw from the regular Google index. Different paths, same starting point: a crawler requests your page and either gets it or gets turned away. (If the acronyms are still fuzzy, our plain-English guide to GEO, AEO and SEO sorts them out.)

The five checks below tell you which side of the door you're on. You need a browser, your website address and about fifteen minutes. No logins, no software, no jargon you can't handle.

Check 1: read your robots.txt

Type your domain into a browser and add /robots.txt to the end, like yoursite.com/robots.txt. Every site has this address, and it's public by design. What loads is a plain text file of instructions that well-behaved crawlers read before touching anything else on your site.

You're scanning for two things. First, a blanket block: a line reading "User-agent: *" followed by "Disallow: /" shuts out every compliant crawler, AI or otherwise. Second, blocks aimed at specific AI crawlers by name. Use Ctrl+F (Cmd+F on a Mac) and search for each of these:

• GPTBot is OpenAI's training crawler. A Disallow here tells OpenAI to stop collecting your pages for model training. On its own it does not remove you from ChatGPT search; that job belongs to the next name.
• OAI-SearchBot is the retrieval crawler behind ChatGPT search. Disallow it and your pages drop out of the pool ChatGPT draws on when someone asks it to find, say, a pool builder in Naples.
• ChatGPT-User fetches pages live when a person asks about them mid-conversation. Block it and ChatGPT has to tell that person it can't open your link.
• PerplexityBot handles Perplexity's live retrieval. Perplexity builds answers from pages it fetches in the moment and lists them as numbered sources. Blocked pages can't be sources.
• Google-Extended is a control switch rather than a crawler you'll see in your logs. It governs whether Google may use your content for Gemini training and grounding. Blocking it does not touch your classic Google rankings and does not decide AI Overviews eligibility; both of those ride on your normal Google indexing.

Depending on who set up your site, you may also see other names in a block list, ClaudeBot, CCBot and Bytespider among them. You don't need to know who operates every crawler to read the file; the rules below apply to any user agent you find.

Reading the results: a "Disallow:" line with nothing after it means that crawler is allowed everywhere. "Disallow: /" means your whole site is off limits to it. No mention of a crawler at all usually means it falls under whatever the "User-agent: *" rules say.

Now the part that surprises owners. You may be blocking these crawlers without ever having decided to. Some CDNs and security platforms write managed blocks into robots.txt on your behalf, through a default or a one-click setting somebody flipped long ago. Cloudflare, for example, can serve a managed section that disallows a long roster of AI crawlers and appends a content signal telling AI systems not to train on your pages. If you find a block like that and you never chose it, don't panic. It's a setting, not a sentence, and check 2 shows you where it lives.

Check 2: open your CDN or security dashboard

robots.txt is a posted sign. The major crawlers respect it, but it only asks. Your CDN or firewall is a locked door: it can refuse the request outright, and no amount of welcoming robots.txt language matters if the server answers 403 Forbidden before a crawler reads anything.

If your site sits behind Cloudflare, log in and review the bot settings. Cloudflare ships AI crawl controls that can block AI bots at the network edge, plus broader bot-fighting modes and WAF rules that challenge anything that doesn't behave like a human browser. Other CDNs and security plugins have their own versions. While you're in there, open the firewall event log. If GPTBot, PerplexityBot or ClaudeBot are stacking up in the blocked column, you have your answer.

To be clear, these settings are legitimate. Blocking AI training on your content is a real business decision some owners make with open eyes. The problem is when it isn't a decision: when a default quietly turns away the systems your future customers ask for recommendations. One telltale sign of a server-level block: a page opens fine in your browser, but a non-browser tool requesting the same address gets an error like 403 Forbidden. When that happens, the front door is closed no matter how welcoming your robots.txt reads, and check 3 gives you a tool-free way to see it from the outside.

Check 3: ask the assistants directly

Now test from the outside. Open ChatGPT and paste something like: "Open this URL and quote its main headline word for word: https://yoursite.com/services/." Then run the same prompt in Perplexity. Pick a specific interior page, ideally one you've updated recently, rather than your homepage.

Three possible outcomes:

• It fetches and quotes accurately. Pass. Retrieval works end to end, from their crawler through your server to your actual copy.
• It says it can't access the page. Fail. Something from check 1, 2 or 4 is standing in the way, and now you know it matters in practice.
• It confidently describes the page but gets details wrong, or describes an old version. Caution. The assistant is answering from memory instead of fetching. This is exactly why you ask for an exact quote: a vague summary can be stitched together from training data, but your current headline, word for word, cannot.

In Perplexity, also glance at the numbered citations under the answer. If your own domain shows up as a source for a question about your own page, live retrieval is working.

Check 4: the view-source test for JavaScript dependence

Right-click an important page on your site and choose "View Page Source." You're now looking at the raw HTML your server sends before the browser runs any scripts. Search it for a sentence you know appears on the page: your headline, a service description, your phone number.

If your copy is there, pass. If the source is a thin shell of script tags and your words are nowhere in it, your site depends on JavaScript to display its content. Google's crawler executes JavaScript and will still see the page eventually. As of mid-2026, most AI crawlers do not. They read the HTML the server hands them and move on. To them, a JavaScript-only page is an empty room.

We see this most often with single-page site builders and with content trapped inside embedded widgets: the booking calendar on a charter site, a menu embed, a review carousel. If the words your customers care about only exist inside a widget, AI readers likely never see them.

Check 5: look for llms.txt, the bonus signal

Last one, and it's extra credit rather than pass-fail. Visit yoursite.com/llms.txt. If a plain text page loads with a short, organized guide to your site, you have an llms.txt file. If nothing loads, no harm done; the absence costs you nothing today.

llms.txt is a voluntary proposal, documented at llmstxt.org: a plain-text map of your site written for AI systems, pointing them to your most useful pages. No engine has committed to honoring it, so treat it as cheap insurance and a tidy crawlability signal, never a ranking lever. We publish one on our own site, and it took minutes, not days. For the full picture of what goes in the file and whether it's worth your time, read our llms.txt guide.

What to do with a failed check

Checks 1 through 4 are the ones that matter; check 5 is a bonus. Here's the triage:

• Failed check 1 or 2 (blocks in robots.txt or at the CDN): usually fixable in under an hour. Our guide to unblocking AI crawlers walks through the exact edits, including how to lift a Cloudflare managed block without dropping protections you actually want.
• Failed check 3 while 1 and 2 pass: look harder at check 4, or at aggressive firewall rules that only trigger on non-browser traffic. The assistants failing to fetch is a symptom; the cause is always upstream.
• Failed check 4: this is an architecture problem, not a settings problem. The fix is getting your copy into the HTML your server actually sends, which usually means changing how the site is built rather than flipping a switch.

If you'd rather hand the whole thing off, auditing and fixing AI crawler access is a productized service for us: llms.txt and AI crawlability.

One honest caveat before you celebrate a clean sweep: passing all five makes your site readable, not recommended. Readable is the floor. Getting cited when a Naples homeowner asks ChatGPT for a seawall contractor, or a Bonita Springs visitor asks Perplexity for a med spa, takes entity clarity, answer-shaped pages, reviews and time. That's the slower work behind our AI search optimization service, and we set a 90-day ramp expectation with every client because honest movement takes that long, long-tail questions first.

Or let our free checker run the access tests

Our free AI Visibility Checker automates the crawl-access portion of this diagnostic. Give it a URL and it reads your robots.txt for the AI crawler names above, tests whether your pages answer requests the way AI systems make them, and hands you a plain-English readout. It can't log into your Cloudflare dashboard or view-source your pages for you, so keep checks 2 and 4 on your own list.

And if the results raise more questions than answers, request a free audit. We run all five checks plus the classic local SEO fundamentals and send the report within one business day, whether you're a plumber in Golden Gate or a dental practice with two offices. No obligation, and you'll at least know whether your showroom door is open.

Frequently asked questions