The agentic web

The last module is the least settled, and it is here because the daily feed keeps returning to it. The question is what changes when the thing arriving at your site is not a person and not a search crawler, but software acting for someone.

Crawlers you now have to decide about

The web has many more automated fetchers than it did: search crawlers, AI training crawlers, AI retrieval crawlers that fetch a page because a user asked something right now, and agents acting on a user's behalf. They have separate user agents and separate robots.txt tokens, and they are a genuine policy decision rather than a technical one.

Blocking training crawlers while allowing retrieval crawlers is a coherent position: you decline to be training data but stay answerable in live products. Blocking everything protects content and removes you from surfaces that increasingly mediate discovery. There is no consensus answer, and reasonable publishers have gone in opposite directions.

The scale is no longer marginal. During the week of 24–28 August 2026, machine traffic passed human traffic on the open web.

From pages to interfaces

The more structural shift is agents that act rather than read. WebMCP is the current attempt at a standard here, and OpenAI, Shopify and Cloudflare are shipping it in live products: sites declare what an agent can do — search a catalogue, check availability, start a return — rather than agents inferring actions from rendered UI.

If that takes hold, part of what you optimise stops being a page and starts being a declared capability. That is a different discipline from technical SEO, closer to API design, and it is far too early to reorganise around.

What is actually actionable today

• Know your bot traffic. Which crawlers fetch you, how much, and at what cost.
• Make a deliberate robots.txt decision about AI crawlers rather than inheriting one.
• Keep answers extractable. Clear, self-contained, well-structured content serves agents for the same reason it serves AI Overviews.
• Watch, do not rebuild. Nothing here yet justifies re-platforming.

What trips people up

• Treating all bots as one thing. Blocking indiscriminately can remove you from search.
• Buying "agent optimisation" services for a standard that is barely deployed.

You have got this when

You can state your site's position on AI crawlers as a deliberate choice with a reason — and you are watching WebMCP without acting on it yet.