AI Crawlers and Publishers: Block Them, or Profit From Them?

For publishers it is the most uncomfortable topic of the past two years. Language models learn and answer from content somebody wrote and paid for – and the traffic that used to come back for it is shrinking. Yet the question “to block or not” has no single correct answer; it has several options with different consequences.

How unequal the ratio is

Cloudflare’s 2025 data gives a frame worth remembering: while with Google roughly 14 downloaded pages fall on one referred visit, with Perplexity it is in the hundreds, with OpenAI approximately a thousand and with Anthropic tens of thousands. In other words: the classic search engine was a trade; this, so far, is mostly a one-way take.

Add the data on reader behaviour: according to Pew Research, with an AI overview shown, a user clicks a classic result in only 8% of cases versus 15% without it – and a link inside the overview in only 1%. Zero-click hits media harder than anyone else, because their product is exactly the information the overview provides.

Distinguish whom you are actually dealing with

A blanket block of “AI bots” is a blunt tool, because they are not one group:

• Training bots (e.g. GPTBot, ClaudeBot) collect data for model training. They practically never return a visit.
• Search bots with citations (e.g. OAI-SearchBot, PerplexityBot) serve answers in which your medium is named and linked. The traffic is small, but the mention exists.
• On-demand bots (e.g. ChatGPT-User) fetch a page because a user just asked for it. That is closest to an ordinary visit.

Each of them can be controlled separately in robots.txt. A sensible default position for most media: consider the training bots, let the search and on-demand ones through. Removing yourself from the answers with citations means disappearing from the place where part of the audience today forms its idea of which media are worth reading.

Three strategies that make sense today

1. Selective access. Open up to search bots, restrict the training ones – especially for the archive and exclusive content. Low risk, no loss of visibility.
2. Paid access. Infrastructure players (Cloudflare and others) today offer modes where the crawler either pays or does not get through. For a medium with a solid volume of content it is a real path to turning the take into income.
3. Licensing. Direct deals with model operators. Available mainly to large publishers, but smaller titles can act jointly within industry associations.

Why being a cited source pays

Traffic is not the only currency. When your medium repeatedly appears as a source in AI answers, its perceived authority grows – and that is exactly what advertisers buy from you. A title the models refer to is more valuable to PR article buyers than a title with the same traffic that is missing from the answers. Citability can even be documented: put queries from your field to the models and record how often you are listed as a source. It is an argument for a sales negotiation that few use so far.

What to do this week

1. Look into your logs at how much traffic the AI bots really take from you – deciding without numbers goes badly.
2. Go through robots.txt and split the bots into the three groups above instead of one blanket rule.
3. Verify whether the models mention you on the topics you build your brand on.
4. Add this figure to your advertiser deck next to traffic and DA.

And then it is only about the buyers who appreciate this learning about your title. That is exactly why it makes sense to have the medium visible where advertisers and agencies pick media – for example on the Linketica marketplace, where you manage the parameters, rules and price list yourself.