On 1 July 2025 Cloudflare changed a default. Every new domain on its network now blocks known AI crawlers unless the owner switches the setting off. Cloudflare sits in front of roughly a fifth of the public web, so a meaningful share of sites launched since that date are invisible to the crawlers behind ChatGPT, Claude and Perplexity. Most of the companies running those sites have no idea.
That is the uncomfortable thing about AI visibility in 2026. The decisions that determine whether an assistant can see your company are rarely made by anyone. They are inherited from a template, a hosting default, a security plugin or a robots.txt file written years ago for a different reason. The result is a growing class of businesses that have, without intending to, opted out of the channel their buyers are moving to.
Why this matters more than it used to
Two years ago, being absent from AI answers cost almost nothing, because few buyers used them for anything serious. That has changed. Research from 6sense found that 94% of B2B buyers now use large language models at some point in the purchasing process. Assistants are where shortlists form, where regulatory questions get answered, and where “which tools handle this” gets asked before anyone opens a search engine.
If an assistant cannot fetch your pages, it cannot cite them. It will answer the buyer’s question anyway, using whatever sources it can reach. Those sources belong to your competitors.
“The strange part is that this is not a hard problem,” says Vasiliy Rusakov, Head of SEO at NeuroSEO, a European agency working with software and technical B2B companies. “Nobody is asking you to outsmart a model. You just have to let it in and give it something readable. Most sites fail at the first step and never find out.”
Three ways a site goes dark without anyone deciding
In practice, AI invisibility comes from three sources, and they compound.
The CDN or security layer. Cloudflare’s default block is the best known example, but it is not the only one. Bot-protection services increasingly classify AI crawlers as unwanted traffic and challenge or block them. The site owner sees nothing wrong, because human visitors are unaffected. The crawler sees a challenge page or a 403, logs the failure, and moves on.
The robots.txt file. Many sites added a blanket block on GPTBot in 2023, when the only conversation about AI crawlers was about training data. That rule is still there, and it has quietly acquired a side effect nobody planned. OpenAI now operates separate crawlers for separate purposes: GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search results, and ChatGPT-User fetches a page when a user asks about it directly. A site that blocked everything OpenAI operates, by pattern or by habit, is out of ChatGPT search as well as out of the training set. Those are different decisions, and most companies only meant to make one of them.
The rendering layer. Google executes JavaScript before indexing a page. Most AI crawlers do not. A crawl-log study published by Vercel with MERJ in December 2024 found that none of the major AI crawlers executed JavaScript. GPTBot and ClaudeBot fetched JavaScript files in 11.5% and 23.8% of their requests respectively, then ran none of them. For a site built as a single-page application, the product description, the pricing and the documentation may exist only after a script runs. To an AI crawler, the page is an empty shell with a navigation bar.
The distinction that changes the decision
Much of the public argument about AI crawlers is framed as a single choice: let AI companies take your content, or shut them out. For most businesses that framing is wrong, because the crawlers do different jobs and can be treated differently.
Training crawlers collect material that may end up in a model’s weights. Blocking them is a legitimate choice, particularly for publishers whose content is the product. Search and retrieval crawlers fetch pages to answer a live question and cite the source. Blocking those removes you from the answer. A B2B company that sells software, services or expertise usually wants the second kind of visibility very much, and has no strong reason to refuse the first either.
Google’s version of this split is easy to misread. Google-Extended is not a crawler at all. It is a robots.txt token that controls whether content Googlebot has already collected may be used for Gemini. It has no effect on Google Search, and AI Overviews are part of Search, so blocking Google-Extended does not remove a site from them. Sites that block it expecting to disappear from AI Overviews have changed nothing except their training-data policy.
A useful way to decide is to write down, for each crawler, what it does and what you want from it:
- Training crawlers (GPTBot, ClaudeBot): your content may inform a model’s general knowledge. Allow if you want the model to know your category well; block if your content is the product.
- Search and retrieval crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot): your pages can be fetched and cited in live answers. Almost every B2B company wants these allowed.
- User-triggered fetchers (ChatGPT-User): a page is retrieved because a person asked about it. Blocking this means a prospect who pastes your URL gets nothing back.
- Policy tokens (Google-Extended): no crawling at all; a switch for how already-collected content may be used.
A thirty-minute audit anyone can run
None of this requires a developer, though a developer will do it faster. The agency’s published guide to AI crawler optimization walks through the full process; the short version fits in a lunch break.
- Read your own robots.txt. Open yoursite.com/robots.txt and search for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended. For each one you find, ask who added the rule and why. If nobody knows, it was not a decision.
- Check the CDN and security settings. If the site sits behind Cloudflare or a similar service, find the AI crawler controls and read what they are set to. Cloudflare separates training bots from other categories; a default created after July 2025 is likely to be a block.
- Fetch a page the way a crawler does. From a terminal, request a key page with the user agent set to GPTBot and read the raw HTML that comes back. If the body is nearly empty and the content appears only in the browser, the rendering layer is the problem.
- Look at the server logs. Search for the crawler names from step one and check the response codes. A healthy picture is a steady stream of 200 responses with full page sizes. Repeated 403s, challenge pages or 402s mean something upstream is turning the crawlers away.
- Ask the assistants directly. Put the question a buyer would ask into ChatGPT, Perplexity and Gemini and see whether your company appears and whether any of your pages are cited. Do it several times, because the answers vary.
Step five is the one that gets attention in the boardroom. The first four are the ones that explain the result.
What to do with what you find
The fixes are unglamorous. Unblock the search and retrieval crawlers you want to be cited by, and make that an explicit rule in robots.txt rather than an absence of rules. Change the CDN setting from a default to a choice. Where content only exists after JavaScript runs, serve a server-rendered or statically generated version of the same page. That last one is development work, but routine development work, and it helps Google too.
Then comes the part that is less mechanical. A crawler that can reach a page still has to find something worth quoting. Models extract passages, not pages. Content that answers a question directly in its first sentences, states facts the company can stand behind and keeps the same claims consistent across the site, review platforms and the trade press is what gets cited. Marketing copy that builds slowly to a vague promise is fetched successfully and ignored.
There is also a reasonable case for not doing any of this. A publisher whose revenue depends on readers arriving at the page, rather than on being named as a source, may decide that AI answers are a channel that pays nothing and takes something. Cloudflare’s pay-per-crawl experiment, which returns a 402 response until a crawler agrees a price, exists for exactly that position. The point is not that everyone should open the doors. The point is that it should be a decision, taken by someone who knows what the doors are.
The companies that benefit most
The businesses with the most to gain from fixing this are not the ones with the biggest marketing budgets. They are the ones whose buyers ask assistants specific, technical questions: software vendors, compliance and regulatory specialists, engineering and scientific suppliers, anyone selling into a sector where the purchase is researched for months. In those markets, an assistant that names three vendors has done most of the shortlisting, and a fourth vendor it could not read never existed.
“We audit sites that have spent years and real money on content, and the content is good,” Rusakov says. “Then we check the logs and the AI crawlers have been getting a 403 since the day a security plugin was installed. All that work, and the systems their buyers now use have never read a word of it. That is the thing to fix first, before anyone writes another article.”
About NeuroSEO
NeuroSEO is a boutique search agency serving English-speaking SaaS, AI and technical B2B companies across Europe. The team of more than thirty senior specialists handles strategy, content, technical SEO, frontend development and DevOps in-house, and runs search and AI-visibility programmes as one piece of work. The agency does not promise rankings or citations; it commits to the work, the sequence and honest reporting.