Claude doesn't crawl the web: it searches Brave's index, and that changes who you need to please

Claude doesn't have its own index of the web. When it searches, it asks someone else. And for anyone wondering whether their site shows up in AI answers, that matters more than almost everything written about GEO — optimising your site so that AI search engines (ChatGPT, Perplexity, Gemini, Google's AI overviews) cite it.

What happened

On 19 March 2025, Anthropic updated the subprocessor list in its trust centre and added Brave Search. A subprocessor is, in plain terms, an outside supplier handling data on the company's behalf: being on that list is official documentation, not a rumour.

Two days later, on 21 March 2025, Simon Willison wrote it up and checked it himself in this post on his blog. He asked Claude for pelican trivia; the model fired off a search and came back with ten cited sources. Willison ran the same query on Brave and got the same list, in the same order. To top it off, when he poked at the internal definition of the model's search tool, the parameter block carried a rather telling label: BraveSearchParams.

Two kinds of proof, and they're worth separating. The subprocessor list is documentary evidence from Anthropic itself. The side-by-side comparison is empirical evidence from a third party, reproducible by anyone. Neither is a press release from someone selling tools.

There's a third fact that isn't news but explains the headline: Brave sells access to its index as a product, through its search API. It's not a back-room deal, it's a catalogue with public pricing.

Why it matters

When a client tells me "I want to show up in Claude", what they're actually asking for is to be in the index Claude consults. Those are two different doors and only one depends on the model.

• The first door is the index. If your site isn't there, no amount of copywriting, schema or llms.txt will save you. The model can't cite what its search engine never shows it.
• The second door is the model, picking two or three of those results to build the answer. That's where content, clarity and actually answering the question come in.

The practical upshot is awkward for old-school SEO: "only Google matters" stops being true the moment part of your traffic starts arriving via a chatbot's recommendation. Google still rules on volume, but the index feeding an assistant may be a different one, and you won't know unless you look.

What I'd do differently from today, without spending a penny:

1. Check whether your site appears in indexes that aren't Google. A site:yourdomain.com on Brave and on Bing takes thirty seconds and tells you whether you exist for them.
2. Read your robots.txt slowly. I keep finding inherited rules that allow Googlebot and block everything else "just in case". Five years ago that cost nothing. Today it locks you out of the first door.
3. Look at the WAF and the CDN, not just robots.txt. Plenty of blocks don't live in a text file: they live in a Cloudflare rule that cuts by country, by volume or by unknown agent.
4. Tell crawler and index apart. One thing is the robot that builds an index, another is the one fetching a page at answer time. Checking which AI crawlers can read each section, and whether your business comes up when several models are asked about your sector, is a half-hour diagnosis; we handle it with the "AI crawlers" and "Does AI cite you?" tools in SeoTres, but the order of the steps is the same whatever you use.

What doesn't change

A line on a subprocessor list from March 2025 isn't a marriage. Suppliers get swapped, added and dropped, and I have no confirmation that this arrangement is identical today: the list is something you check, not something you remember.

Being in the index guarantees nothing either. It's necessary, not sufficient: out of ten results, the model cites two or three. Anyone promising you'll "appear in ChatGPT" is selling a probability dressed up as a certainty.

And the headline about Brave "selling its index to AI" is true but hardly a scandal. Licensing a search index is an old, perfectly ordinary business. What's new is who's buying and what for.

Our take

What I like most about this story isn't the fact itself, it's that it could be verified from the outside with a search about pelicans. That's what's missing from 90% of what gets published on GEO: checks anyone can repeat on their own laptop, instead of theories about what models supposedly "prefer".

My position: nearly everything sold as AI optimisation is make-up over something boring and checkable. Being indexed somewhere other than Google. Serving readable HTML without needing three megabytes of JavaScript to show a paragraph. Not accidentally blocking the people trying to read you. That doesn't fill a four-hundred-slide course, but it's the only thing that keeps working when the supplier underneath changes.

And it will change. Any tactic tied to one specific search engine behind one specific model ages terribly; we saw this with Google's featured snippets, when half the industry rewrote their pages for a box that later mutated. What doesn't age is being crawlable and answering a question well.

One warning from the systems side, which is where we actually run into this: the block that shuts you out is almost never a marketing decision. It's a rule someone added to the firewall three years ago to stop a scan, and nobody has reviewed it since. Checking that costs a morning and is worth more than rewriting your homepage copy.

Any questions, tell me and we'll look at it.

All the best, Vicente.