We have spent the last decade complaining about “Crawled - currently not indexed.” We treat it as a failure state. We treat it as a bug.

But in the Agentic Web of 2025, “Indexation” is not the goal. “Retrieval” is the goal.

And paradoxically, to maximize Retrieval, you often need to minimize Indexation.

The Information Density Argument

LLMs (Large Language Models) and Search Agents operate on Information Density. They want the highest signal-to-noise ratio possible.

For the last decade, the mantra of content marketing has been “Long-Form Content.” Creating 3,000-word “Ultimate Guides” was the surest way to rank. But as the consumers of content shift from bored humans to efficient AI agents, this strategy is hitting a wall. The new metric of success is Information Density.

The Context Window Constraint

While context windows are growing (128k, 1M tokens), they are not infinite, and more importantly, “reasoning” over long context is expensive and prone to “Lost in the Middle” phenomena.