The ultimate form of “white hat cloaking” is Content Negotiation. It is the practice of serving different file formats based on the requestor’s capability.

HTTP Accept Headers

If a request includes Accept: application/json, why serve HTML?

  • Human Browser: Accept: text/html. Serve the webpage.
  • AI Agent: Accept: application/json or text/markdown. Serve the data.

The “Headless SEO” Approach

This approach creates the most efficient path for agents to consume your content without navigating the DOM. Instead of forcing the agent to:

  1. Download HTML.
  2. Parse CSS.
  3. Execute JS.
  4. Extract Text.
  5. Infer Structure.

You simply hand them the Structure.

Is this Cloaking?

Google’s definition of cloaking warns against serving different content to deceive. If your JSON-LD contains the exact same facts as your HTML, just in a different format, it is not deception; it is Format Optimization.

By serving data-first payloads, you ensure that 100% of your content is ingested, compared to the 60% that usually survives DOM distillation. You are removing friction from the ingestion pipeline.

The Content Negotiation Spec

Standardizing this behavior is crucial. We recommend using the Link header to point to alternative representations.

Link: <https://api.example.com/data.json>; rel="alternate"; type="application/json"

This informs the agent: “If you prefer data, go here.” This is not cloaking; it is HATEOAS (Hypermedia as the Engine of Application State). It is the architecture of the web as Roy Fielding intended, updated for the age of AI.

Glossary of Terms

  • Agentic Web: The specialized layer of the internet optimized for autonomous agents rather than human browsers.
  • RAG (Retrieval-Augmented Generation): The process where an LLM retrieves external data to ground its response.
  • Vector Database: A database that stores data as high-dimensional vectors, enabling semantic search.
  • Grounding: The act of connecting an AI’s generation to a verifiable source of truth to prevent hallucination.
  • Zero-Shot: The ability of a model to perform a task without seeing any examples.
  • Token: The basic unit of text for an LLM (roughly 0.75 words).
  • Inference Cost: The computational expense required to generate a response.