For thirty years, robots.txt has been the “Keep Out” sign of the internet. It was a simple binary instruction: “Crawler A, you may enter. Crawler B, you are forbidden.” This worked perfectly when the goal of a crawler was simply to index content—to point users back to your site. But in the Generative AI era, the goal has shifted. Crawlers don’t just index; they ingest. They consume your content to train models that may eventually replace you.