How to unblock AI crawlers (and whether you should)

This guide is the fix-it half of a pair. If you are not sure yet whether AI crawlers can reach your site, start with our diagnostic walkthrough and come back once you know what is actually broken. Everything below assumes you found a block and want to remove it on purpose, which is exactly the right way to do it.

The tradeoff, stated before any fixes

Unblocking AI crawlers means AI systems can read your content, quote it, summarize it, and use it to answer other people's questions. Someone might get the gist of your water heater page from a chat answer without ever visiting your site. That is the deal, and we are not going to pretend it costs nothing.

For a local service business, it is usually a good deal anyway. You did not publish your service pages so strangers could admire the writing. You published them so a homeowner in Naples with a failing water heater would call you. If an AI assistant reads that page and tells the homeowner your company handles replacements in Collier County, the content just did its job. The page was always bait. The customer was always the catch. Trading content for customers is the whole business model, and AI answers are one more place that trade happens. Our guide on how AI chooses which businesses to recommend covers what happens once the door is open.

Before you touch a single setting, understand the one distinction that splits this decision in two:

• AI training means a company uses your content as raw material to build its models. Your words become ingredients. It is permanent, invisible, and sends you no customers directly.
• AI input, sometimes called retrieval or grounding, means an assistant fetches your page at the moment a real person asks a question, uses it to answer, and can point that person to you. This is where the near-term leads come from.

You can allow one without the other. Blocking GPTBot while allowing OAI-SearchBot tells OpenAI, in effect: do not train on my content, but do show it to people searching in ChatGPT. Some CDN-managed robots.txt files express the same split with a content signal line such as ai-train=no. For most local service businesses, allowing both makes sense, because exposure is the point, but the split is real and choosing it is legitimate.

How sites end up blocked without anyone deciding

Most blocked sites never chose to be blocked. Three things do the deciding, and none of them is the owner:

• CDN managed rules. Cloudflare and similar services offer one-click controls that block AI crawlers across the board. Sometimes the toggle got flipped during setup by whoever built the site. Sometimes a managed block appears in robots.txt that nobody on your team wrote. Either way, the block is live and speaking for you.
• Security plugins. WordPress firewall and security plugins often bundle bot protection that treats every non-browser visitor as a threat. GPTBot gets the same greeting as a vulnerability scanner: a 403 and a closed door.
• Old scraper paranoia. Plenty of robots.txt files were written years ago to fight content thieves and copycat sites. Blanket disallow rules from that era now catch AI crawlers that did not exist when the rule was written.

In each case, nobody weighed the tradeoff above. A default did. The rest of this guide replaces that default with a decision.

Step 1: fix robots.txt

robots.txt is a plain text file at the root of your domain, at yourdomain.com/robots.txt. It is a posted policy, not a lock: well-behaved crawlers read it and obey it, so it is the first thing to fix and the easiest. The clearest approach is an explicit allow block for each AI agent you want to welcome, so there is no ambiguity about your intent.

Here is the shape of what we run in our own robots.txt at naplesseo.com, one block per crawler:

• User-agent: GPTBot
  • Allow: /
• User-agent: OAI-SearchBot
  • Allow: /
• User-agent: ChatGPT-User
  • Allow: /
• User-agent: PerplexityBot
  • Allow: /
• User-agent: Google-Extended
  • Allow: /

What each of those agents actually is, as of mid-2026:

• GPTBot is OpenAI's training crawler. It gathers content that may end up in future models. This is the ai-train side of the split.
• OAI-SearchBot handles retrieval for ChatGPT search. If you want to be found and cited when people search inside ChatGPT, this is the one that matters most.
• ChatGPT-User fetches pages live when someone in a ChatGPT conversation asks about them. Blocking it means the assistant cannot read your site even when a user asks it to directly.
• PerplexityBot does live retrieval for Perplexity, which shows numbered source citations. Allowing it is how your pages become one of those sources.
• Google-Extended governs whether Google may use your content for Gemini training and grounding. It does not control classic Google indexing, and it does not decide AI Overviews eligibility, so allowing it will not touch your normal rankings either way.

Two gotchas while you are in the file. First, look for a leftover blanket rule such as User-agent: * followed by Disallow: /. That single pair overrides good intentions everywhere else. Second, if your CDN manages robots.txt for you, hand edits may be silently overwritten or added to, which brings us to step 2.

Step 2: review your CDN's AI crawl controls

If your site sits behind Cloudflare or a similar service, there is likely a dashboard control governing AI crawlers separately from anything in your files. Cloudflare in particular offers managed AI crawl settings that can block a long roster of agents at the edge and inject a managed block section into your robots.txt automatically. If you have ever seen crawler names in your robots.txt that you know you never typed, this is usually where they came from.

Log in, find the AI crawl control or bot management area, and look at what the toggle is actually set to. Then decide it deliberately. If your call, after the tradeoff section above, is to welcome AI crawlers, switch the setting to allow and confirm the managed lines disappear from your robots.txt. If your platform lets you choose per crawler, or split training from search retrieval, even better, because that maps exactly onto the ai-train versus ai-input distinction.

The point is not which way you flip it. The point is that a person who understands the tradeoff flips it, instead of a default nobody remembers setting.

Step 3: fix the firewall rules that 403 crawlers anyway

Here is the sneakiest failure, and the one most likely to survive a fix that only touched robots.txt: robots.txt says allow, and the server says 403 Forbidden. The crawler checks your policy, sees a welcome, requests the page, and a web application firewall or bot-fight rule fingerprints it as automated traffic and refuses it. Every robots.txt checker on earth will tell you the site is open. It is not.

This failure hides because the two systems never talk to each other. robots.txt is a note on the door. The WAF is the bouncer, and the bouncer does not read the note.

Places to look:

• WAF custom rules that challenge or block requests by user agent, especially anything matching the word bot.
• Bot-fight or super-bot-fight style modes on your CDN, which challenge automated traffic wholesale. AI crawlers cannot solve a challenge page, so a challenge is a block.
• Security plugin firewalls at the application level, which often ship with aggressive bot lists switched on.
• Host-level bot protection that some managed hosting providers apply before traffic ever reaches your site.

The fix is to add explicit skip or allow rules for the AI crawlers you welcomed in step 1, or to use your platform's verified-bots allowance where one exists. Then test the honest way: fetch a page from your site while sending an AI crawler's user agent string, and confirm you get a 200 response with real HTML back, not a 403 or a challenge page. The diagnostic guide walks through exactly how to run that test.

Step 4: verify, then keep watching

Unblocking is not done when you save the settings. It is done when you see AI crawlers actually reading the site again. Verify in three passes:

• Re-run the access checks. Fetch robots.txt and confirm your allow blocks are live and no managed block lines came back. Fetch key pages with AI user agents and confirm 200s across the board.
• Watch your server logs. Over the following days and weeks, look for GPTBot, OAI-SearchBot, ChatGPT-User, and PerplexityBot showing up in your access logs. Returning crawlers are the ground truth that the door is really open.
• Check the visibility side. Our free AI Visibility Checker shows how your business currently surfaces in AI search, which gives you a baseline to measure against.

Set expectations honestly. Crawlers come back on their own schedule, and citations in actual AI answers build more slowly than log entries do. We set a 90-day ramp expectation with every client for a reason, and long-tail questions usually move before the big ones. If the logs stay silent for weeks, re-test the 403 scenario from step 3, because a quietly regenerating firewall rule is a classic relapse.

Then put out the welcome mat: llms.txt

Once crawlers can get in, you can go one step further and tell them where to look. llms.txt is a plain text map of your site written for AI systems, a voluntary proposal from llmstxt.org. No engine has committed to honoring it, so treat it as cheap insurance and a crawlability signal rather than a ranking lever. We publish llms.txt and llms-full.txt on our own site because the cost is close to zero. The full explanation, format and all, lives in our llms.txt guide.

When you should not unblock

This guide has an audience, and it is local service businesses: plumbers, roofers, dentists, dock builders, med spas. For those businesses, content exists to bring in calls, so letting AI systems read and repeat it is usually the point.

If your content is the product, the math flips. Paywalled publications, photographers licensing images, course creators, research firms selling reports: for these businesses, an AI answer that reproduces the substance of the work is a real cost with no offsetting phone call. Blocking training crawlers, limiting retrieval, or blocking everything is a defensible business decision there, and plenty of content businesses have made it with their eyes open.

The failure mode we care about is not blocking. It is blocking by accident. A Naples roofer whose CDN quietly turns away every AI crawler is not protecting anything. That roofer is just invisible to a growing set of systems people now ask for contractor recommendations. Whichever way you go, go there on purpose.

Want it done for you?

We audit AI crawler access, fix the robots.txt and firewall layers, write the llms.txt, and verify the crawlers return, as part of our llms.txt and crawlability service. It is one slice of the AI search optimization work we include in every plan. And if you would rather do it yourself, everything above is genuinely enough, and the diagnostic guide will tell you whether it worked.

Frequently asked questions