An AI Overview once reported that the cat on tamethebots.com held a job title. The cat’s name was Odd. The role was invented. The citation pointed to a plain-text file with no schema, no author field, and no inbound links — a file that had been live for roughly two weeks. That detail circulated widely. Less widely circulated: whether anyone else could reproduce it, where it came from, and what, precisely, it proves.

Imagine a rate card that does not yet exist. For $250 per month, your product appears in the recommended-tools section of a documentation site’s llms.txt, which is parsed by coding agents at the moment a developer asks what library to use. The agent fetches the file, reads the description string, and your product lands in context at the exact instant the decision is made. This is a coherent-sounding product. Adjacent versions are already being sold: paid brand mentions in AI Overviews, sponsored inclusions in AI-generated listicles, and placement in curated “agent context” resources. DiNardi’s audit in Search Engine Land found placements at $250 per mention, with PBN-style inventory selling at 10–15x backlink rates. The rate card for llms.txt is merely the logical next step.

XML Sitemaps have been a staple of SEO for two decades. However, LLMs and AI agents ingest data differently than traditional crawlers. The scale of ingestion for training runs (e.g., Common Crawl) requires a more robust approach. The Importance of lastmod For AI models, freshness is a critical signal for reducing perplexity and preventing hallucinations. A sitemap with accurate, high-frequency lastmod tags is essential. It signals to the ingestion pipeline that new training data is available.

While robots.txt tells a crawler where it can go, llms.txt tells an agent what it should know. It is the first step in “Prompt Engineering via Protocol.” By hosting this file, you are essentially pre-prompting every AI agent that visits your site before it even ingests your content. This standard is rapidly gaining traction among developers who want to control how their documentation and content are consumed by coding assistants and research bots.

In the high-stakes poker game of Modern SEO, llms.txt is the competitor’s accidental “tell.” For two decades, we have scraped sitemaps to understand a competitor’s scale. We have scraped RSS feeds to understand their publishing velocity. But sitemaps are noisy—they contain every tag page, every archive, every piece of legacy drift. They tell you what exists, but they don’t tell you what matters. The llms.txt file is different. It is a curated, high-stakes declaration of what a website owner believes is their most valuable information. By defining this file, they are explicitly telling OpenAI, Anthropic, and Google: “If you only read 50 pages on my site to answer a user’s question, read these.”

HTML is for browsers; Markdown is for brains. LLMs are trained heavily on GitHub repositories, StackOverflow, and technical documentation. This makes Markdown their “native” format. They “think” in Markdown. Token Efficiency Markdown is less verbose than HTML. HTML: <h1>Title</h1> (9 characters, ~3 tokens). Markdown: # Title (7 characters, ~2 tokens). HTML List: <ul><li>Item</li></ul> (21 characters). Markdown List: - Item (6 characters). Across a 2,000 document, this saves thousands of tokens. A clean Markdown file consumes fewer tokens than its HTML equivalent, allowing more content to fit into the context window.

The /llms.txt standard is rapidly emerging as the robots.txt for the Generative AI era. While robots.txt was designed for search spiders (crawling links), llms.txt is designed for reasoning engines (ingesting knowledge). They serve different masters and require different strategies. The Difference in Intent Robots.txt: “Don’t overload my server.” / “Don’t confirm this duplicate URL.” (Infrastructure Focus) Llms.txt: “Here is the most important information.” / “Here is how to cite me.” / “Ignore the footer.” (Information Focus) Content of the File A robust llms.txt shouldn’t just be a list of Allow/Disallow rules. It should be a map of your Core Knowledge.