An AI Overview once reported that the cat on tamethebots.com held a job title. The cat’s name was Odd. The role was invented. The citation pointed to a plain-text file with no schema, no author field, and no inbound links — a file that had been live for roughly two weeks.

That detail circulated widely. Less widely circulated: whether anyone else could reproduce it, where it came from, and what, precisely, it proves.

The Evidence Audit: What Survives Without the Logs

The cats.txt experiment produced four distinct claims. The right way to evaluate the experiment is to ask, for each claim, whether a reader who was not Mark Williams-Cook could have checked it at the time. Running that test produces the following table.

Two consequences worth stating explicitly.

The proof that dropped is the one everyone quoted. Search Engine Journal, DesignRush, Cittago, Something Inc., and the rest all lead with “PerplexityBot, GPTBot and ClaudeBot all fetched it.” Not one of those outlets could have checked that claim. The laundering supply chain Williams-Cook identifies as a problem in GEO ran, in this case, on his own post.

Indexing back-implies one crawl, and only one. For Google to index cats.txt, Googlebot must have fetched it. Nothing in the surviving evidence back-implies that any AI-vendor crawler did. Proof #2 rescues a fragment of Proof #1 — one crawl, by Googlebot — and nothing more.

There Was a Prior Reason for Scepticism

This is the post’s sharpest point, and it comes from data everyone was already citing. Ahrefs’ study of 137,210 domains found that AI bots never requested llms.txt on domains where the file didn’t exist — no probing, no speculative fetching. Among valid llms.txt files, 97% received zero requests in a month.

Against that base rate, a brand-new file at a novel, undocumented path on a low-traffic domain drawing crawl hits from four separate AI vendor crawlers is not the expected outcome. The largest study in the field predicted the opposite of the screenshot, and nobody noticed the tension because the screenshot confirmed what everyone wanted to believe.

Three Claims Being Conflated

The experiment generated a clean-sounding lesson: “arbitrary text files work for AI visibility.” That lesson bundles three distinct claims, which have different evidential statuses.

Claim A: Any text file gets crawled, indexed, and can be cited. Demonstrated for Googlebot and Google’s grounding, trivially — that is the file behaving as a web page, a property it has had since 1996. This claim survives.

Claim B: Therefore arbitrary text files are a visibility lever. Not shown. The retrievals happened on queries with zero competing sources. The AI Overview described Odd because nothing else answered “what cat is on tamethebots.com.” Answer capture is inversely proportional to competing source supply. Nothing about the file extension mattered.

Claim C: Therefore llms.txt does nothing. Not shown by this experiment either. That conclusion comes from the correlation studies — Ahrefs, SE Ranking, OtterlyAI — which are separate and much stronger evidence than one n=1 hoax. The experiment cannot simultaneously prove llms.txt is ineffective and prove that an unlisted cats.txt file is effective; both cannot be true on the same mechanism.

Which Layer Did This Actually Happen In?

Everything Checkable Was Retrieval. None of It Was Training.

Removing the log evidence strengthens this conclusion, because the crawler logs were the only thing in the whole affair that pointed at the training layer at all. Without them, every surviving observation is a retrieval-layer observation.

Four reasons, none of which depend on logs:

Timeline. The file went live in mid-July 2026. ChatGPT endorsed it roughly two weeks later and called it satire by early August. No frontier model pretrains, evaluates, and redeploys on that cadence.

The AI Overview cited a URL. Citation is a grounding artifact by definition. Parametric recall does not emit a source link to a file created weeks earlier.

The flip. “Yes it helps” to “it’s a joke” with no model update between. Weights don’t change because a Substack post circulates; a live retrieval index does.

Williams-Cook says so himself, in the part everyone skipped: the model runs a search, lands on a page that ranks because it’s indexed, and reads it — the file functioning as a web page, not as a standard.

The experiment contains no evidence about training data. Not weak evidence. None.

The Training Pathway Is Open — But You Have to Argue It from Public Sources

The argument for training-layer exposure now has to be rebuilt from things a reader can check, since the logs are gone.

The crawler fleets are documented and split. GPTBot collects training data; OAI-SearchBot builds the ChatGPT search index. Anthropic runs ClaudeBot (training), Claude-SearchBot (search index), and Claude-User (live fetch). Google-Extended is a robots.txt token governing Gemini training and grounding, separate from Googlebot. All of this is vendor-documented and publicly readable.

Training crawlers represent a measurable share of live traffic. Cloudflare’s May 2026 crawler data placed Googlebot at 27.26% of AI-adjacent bot requests, GPTBot at 11.48%, and ClaudeBot at 9.73% — a third-party aggregate, not one person’s screenshot.

Common Crawl is the other door, and it’s auditable. CCBot crawls roughly monthly, honours robots.txt, doesn’t execute JavaScript, and packages responses into WARC archives. A root .txt returning 200 is exactly the shape that lands cleanly in a WARC record. Nearly every major open pretraining corpus — C4, RefinedWeb, FineWeb, RedPajama, DCLM — derives from Common Crawl.

The Common Crawl CDX index is a public, reader-verifiable record. If CCBot captured cats.txt, that’s a timestamped, third-party attestation that at least one corpus-collection crawler fetched the file — without relying on anyone’s private logs.

# Which crawls are available
curl -s https://index.commoncrawl.org/collinfo.json | head

# Did CCBot capture cats.txt?
curl -s "https://index.commoncrawl.org/CC-MAIN-2026-25-index?url=tamethebots.com/cats.txt&output=json"

# Your own llms.txt
curl -s "https://index.commoncrawl.org/CC-MAIN-2026-25-index?url=yourdomain.com/llms.txt&output=json"

A hit is timestamped and re-runnable by any reader. A miss is equally publishable — arguably more interesting given the base rates. Two caveats to print with the result: CC presence is not proof of inclusion in any model’s training set, and CCBot is not GPTBot. Different door, same room.

The Two Layers Have Inverted Cost and Persistence Profiles

The cats.txt affair ran entirely in the left column. Everything that made it look benign — the fast correction, the visible citations, the reversibility — is a property of the retrieval layer, not a property of the tactic.

The Volume Bar for the Training Layer Is Lower Than Assumed

Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that approximately 250 malicious documents could backdoor models from 600M to 13B parameters — a near-constant number regardless of model scale, representing roughly 0.00016% of tokens for the 13B case. 100 documents didn’t reliably work; 250 did. A separate Nature Medicine study found that replacing 0.001% of training tokens with medical misinformation produced models measurably more likely to propagate errors while matching clean models on every standard benchmark.

Mandatory caveats: the Anthropic study involved a narrow, low-stakes gibberish backdoor in models pretrained from scratch at small scale. Anthropic explicitly states the results are unlikely to reflect significant risk in frontier models and that it’s unclear whether the pattern generalises to larger scale or more harmful behaviours. Never read this as “250 documents can poison ChatGPT.”

What survives is the order of magnitude. cats.txt produced low tens of artifacts. A few hundred is a normal content sprint. The relevance is not that the hoax succeeded — it’s that the ceiling on training-layer influence, if it exists, has not been publicly established.

The Open Experiment Nobody Is Running

Every quarter, ask each newly released model about cats.txt with all tools and search disabled. The release where a model describes Odd the Render Cat from parametric memory alone is the date the fabrication entered the weights. If none ever does, the filters worked — also publishable.

This is now the only remaining route to a training-layer answer, since the crawler evidence is gone. The CDX query establishes whether a corpus-collection crawler fetched the file. The longitudinal prompt probe establishes whether any of it survived filters and entered the weights. Running both, and publishing both outcomes, is the experiment the field needs.

Other Conclusions Available

Server logs are the least auditable evidence in GEO, and the field treats them as the most reliable. Every serious llms.txt study — Ahrefs, OtterlyAI, Reboot, Cittago — rests on logs. Logs are private, unattested, unreproducible by any reader, and trivially editable. Meanwhile the evidence types researchers dismiss as weak — index inclusion, AI Overview output — are at least publicly re-runnable. The field’s confidence ordering is inverted relative to its verifiability ordering.

The fix exists and nobody is asking for it: cryptographically verifiable requests. Agents implementing web-bot-auth sign requests under RFC 9421 HTTP Message Signatures. A signed request is log evidence a third party can verify — the signature either validates against the operator’s published key or it doesn’t. Publishing the signature material alongside a crawl claim converts a screenshot into an attestation. Right now, the agents that sign (ChatGPT agent mode, Browserbase, Anchor Browser, Manus, Cloudflare Browser Rendering, Bedrock AgentCore Browser) are not the ones GEO studies count.

It was a successful entity-injection campaign, reported as a negative result. The file did nothing; the campaign worked. A spec page, a LinkedIn article, voluntary adopters, and trade-press pickup. The control condition — the discourse surrounding the file — outperformed the treatment. This conclusion is unaffected by the logs question.

“The models self-corrected” is backwards. Correction required a human debunk to become the dominant framing. No model detected anything. For fabrications nobody debunks, there is no correction path. The likely outcome here — that the correction enters the weights and the fabrication doesn’t — is a lucky accident of this case, not a property of the mechanism.

The four proofs are exactly what the GEO tooling market sells. Bot-hit dashboards productise Proof #1 — the proof that just evaporated. Ahrefs found that roughly 12% of llms.txt requests came from tools that audit, scan, or study llms.txt. The bad evidence persists partly because it is the product.

A pre-hoax artifact is worth noting for its timeline. An agency published “Robots.txt vs LLMS.txt vs Cats.txt” in September 2025 — approximately 11 months before the Williams-Cook experiment — conceding that cats.txt “is not a known standard” and then inventing a comparison table and best-practice guidance for it. This is a verifiable artifact of how the content supply chain manufactures purpose for filenames with no referent. Promote it up your timeline honestly: catstxt.org also appears to predate Williams-Cook’s post, and that timeline remains unresolved.

An indexed llms.txt is an indexation liability nobody prices in. Google offered Search Console verification for cats.txt. Your llms.txt is a rankable, citable, user-facing URL that can be cited instead of the page you wanted cited.

A Protocol Built on Checkable Things

The following tiers order checks by what a reader can reproduce, not by what feels most authoritative.

Tier 1 — Anyone Can Reproduce These

Index inclusion. Run a site: query on the .txt URL; use Search Console URL Inspection; check separately whether the file surfaces for brand queries. Screenshot with a date.

Re-run the AI Overview and the chat prompts. Report what you see today as a fresh dated observation, not a replication. Note locale and personalisation caveats. Google’s platform-specific guidance on llms.txt makes clear that behaviour varies by product.

Parametric vs. grounded check. Ask the same question with search enabled and disabled. Answers that appear only with search on are retrieval. Use the API with no tools; consumer UIs search silently and contaminate the result.

Common Crawl CDX lookup. The queries above establish whether a training-adjacent crawler fetched the file. A hit is a timestamped third-party record. A miss is equally publishable. Check several snapshots before concluding absence. Use the cdx-index-client for batch queries.

Wayback CDX as a second public record. Establishes what the file contained and when, independent of any server.

Longitudinal probe. Same prompt, tools off, each new model release. Log dates. The release where a model responds from parametric memory — without search grounding — is the date to watch.

Tier 2 — Your Own Site; Evidence for You, Not Your Reader

Label this tier honestly when reporting. These are the right experiments; they produce claims your audience cannot check.

Differential fetch test. Publish unlinked: /llms.txt (real), /cats.txt (nonsense, structurally identical), /zzq-control.txt (nonsense, neutral name). Log path, timestamp, user-agent, IP, ASN. The difference between the three is the signal.

Identify bots by verification, not user-agent string. Reverse DNS plus published IP ranges, and check for RFC 9421 Signature / Signature-Input headers where the agent implements web-bot-auth. Publish the verification result, not the log line — that is the actionable ask from the §6.2 finding above.

Bucket every hit by layer, not vendor. Using published crawler reference tables:

Ahrefs found SEO audit tools alone at roughly 21% of llms.txt requests. Failing to strip the tooling tier is how “AI bots crawl it” claims get inflated. Cittago found 79% of AI crawler requests were stopped at the firewall before they fixed it — check the confounder before concluding zero.

Canary strings. A high-entropy nonce plus a fabricated fact in the .txt only; a second nonce in HTML only. Query the assistants and see which returns. Prior art: the canary-token scraper-identification paper on arXiv.

Orphan-URL test. URLs listed only in llms.txt, discoverable no other way. See Reboot’s design.

The placebo. Every GEO test should include a structurally identical nonsense control file. If the placebo gets the same traffic, the signal is null by construction.

What You Cannot Check, Including on Your Own Site

Every claim about the live-fetch layer — ChatGPT-User, Claude-User, coding agents — rests on private logs. There is no CDX equivalent. There is no re-runnable public query. When an llms.txt study concludes that coding agents dominated traffic (Ahrefs found roughly 10% of requests from coding agents against 2% for AI assistants and 1% for retrieval bots, with Claude-Code and GPTBot the top individual fetchers), that finding is real and useful — but it is evidence for the experimenter, not evidence a reader can assess.

This is not an argument against running log studies. It is an argument for labelling them accurately. The field currently presents private-log evidence as the gold standard and treats publicly re-runnable evidence — index queries, AIO screenshots — as anecdote. That ordering is backwards, and the cats.txt affair illustrates exactly why it matters: the claim that dropped first was the one everyone cited most.

The AI visibility landscape has three distinct layers, each with different agents, different latency, and different reader-verifiability:

llms.txt is primarily a layer-3 artifact — it gets read at live-fetch time, when a coding agent or assistant fetches a developer’s documentation. It is sold as a layer-2 lever (retrieval indexation), and the debate about it is conducted with layer-1 intuitions (training data). The evidence mismatch runs all the way down.

The Portable Test

Williams-Cook’s experiment leaves behind one useful instrument, independent of any contested evidence: the portable test for GEO proof.

Would this proof still hold if the page were nonsense — and could my reader check it if it weren’t?

Every claim about AI visibility that fails the first question has a control-condition problem. Every claim that fails the second has an evidence-class problem. The cats.txt affair, stripped of the logs, fails the second question on its most-cited claim. The field repeated that claim anyway. The mechanism that allowed that to happen — private logs laundered through trade press — is the same mechanism that will repeat it on the next experiment, unless the protocol changes.

The CDX query is the protocol change. It takes 15 seconds and produces evidence any reader can rerun. That it has not become standard practice in llms.txt research is the most auditable finding in this entire episode.

References:

• Mark Williams-Cook, “How cats.txt showed llms.txt evidence is GEO astrology” (Substack).
• cats.txt draft spec: markwilliamscook.com.
• A live implementation: tamethebots.com/cats.txt.

Coverage:

• Search Engine Journal
• Something Inc.
• Cittago
• DesignRush

Data:

• Ahrefs
• SE Ranking
• ALLMO
• OtterlyAI
• Reboot

Other references:

• Training-layer research: Anthropic / UK AISI / Alan Turing Institute
• OATML summary
• Common Crawl: about, CDX index docs, index server.
• Crawler fleets: Gupta Deepak reference table
• Cloudflare May 2026 data via GEO Toolbox
• Canary-token method: arXiv.
• Early artifact: SEO Marketing Agency, September 2025.