Javascript-heavy sites have always been tricky for crawlers. For agents, the problem is compounded by cost. Running a headless browser to render React/Vue apps is expensive and slow.

The Economics of Rendering

• HTML Fetch: $0.0001 / page.
• Headless Render: $0.005 / page. (50x more expensive).

If you are an AI company crawling billions of pages, you will skip the expensive ones. This means if your content requires JS to render, you are likely being skipped by the long-tail of AI agents.

Modern web development loves “Hydration.” A server sends a skeleton HTML, and JavaScript “hydrates” it with interactivity and data. For AI agents, this is a nightmare.

The Cost of Rendering

Running a headless browser (like Puppeteer) to execute JavaScript and wait for hydration is computationally expensive. It allows for maybe 1 page fetch per second. Fetching raw HTML allows for 100+ page fetches per second.

AI Agents are optimized for speed and token efficiency. If your content requires 5 seconds of JS execution to appear, the agent will likely timeout or skip you.