Deep dive · Theo Vance (The Builder) · 2026-07-11 · The browser turned into a typed tool surface this week. There are three ways an agent can read a page — and only one of them guesses.

Here is a task you probably gave an agent this week: “add this item to the cart and confirm the total updates.”

Watch how a screenshot-driven agent does it. It takes a picture of the page. It reasons over the pixels, decides the “Add” button is near coordinate (840, 512), and clicks there. It takes another picture to see what happened. Three steps in, it has clicked a tooltip instead of the button, and it has re-read a 2,700-token JPEG of your own UI on every turn to figure that out.

That is the default, and it is the expensive, flaky way. This week the alternative stopped being a research demo. Chrome DevTools for agents is stable — an MCP server plus a token-efficient CLI, 47 tools built on Puppeteer and the Chrome DevTools Protocol. And WebMCP — the browser API that lets a page hand an agent its own tools — shipped a fresh Draft Community Group Report dated 10 July and is in a Chrome origin trial. Put them together and the browser is now a first-class agent runtime.

The useful way to hold all of it: there are three ways an agent can perceive and act on a web page, they cost wildly different amounts, and they fail in different ways. Pick the wrong one and you pay for it in tokens and flake. This is the part worth keeping after the changelog scrolls away.

Level 1: pixels

The agent sees a screenshot and outputs coordinates. This is Anthropic’s Computer Use lineage, and a pile of Puppeteer wrappers.

The cost is not a vibe; it is arithmetic. Claude views an image in 28×28-pixel patches, and each patch is a visual token: ⌈width/28⌉ × ⌈height/28⌉. A 1920×1080 screenshot is 69 × 39 = 2,691 tokens on Opus 4.8, per Anthropic’s own table. Every step adds a frame. A twenty-step task that keeps its frames in context is spending tens of thousands of tokens looking at pictures of a page whose entire HTML would fit in a few thousand.

And the pixels lie to you. Anthropic’s docs say the model’s coordinate outputs are “approximate” and tell you to “verify outputs before relying on them”; that it “might hallucinate or make mistakes when interpreting low-quality, rotated, or very small images.” That is a fair description of a login form. Vision is the reason your browser agent clicked the wrong element and then confidently reported success.

There is exactly one thing pixels are good for: pages that don’t expose their structure at all. A canvas app, a WebGL game, a Figma board. There, the screenshot is the only signal. Keep vision for that. Stop using it as your default.

Level 2: the accessibility tree

The agent gets a structured text snapshot of the page — the same tree a screen reader walks. Every interactive element has a role (button, textbox), an accessible name, a state, and a stable reference. The agent clicks a reference, not a coordinate.

This is what Playwright MCP does with browser_snapshot, and what Chrome DevTools for agents does with take_snapshot. The snapshot looks like this:

- textbox "Email" [ref=e5]
- textbox "Password" [ref=e6]
- button "Sign in" [ref=e10]

The agent calls browser_click(ref="e10"). No guessing where the button is. It’s e10.

The gap from Level 1 is enormous and it runs in your favor on both axes. Playwright reports a snapshot at roughly 200–400 tokens versus thousands for a screenshot — call it a 10× cut on the perception bill (their figure; the ratio depends on page size, but the direction is not in doubt — even Google shipped a CLI for its DevTools agent specifically as a “token-efficient” path). And reliability goes up, not down, because e10 is a handle, not a hypothesis. A reference either resolves or it doesn’t; a coordinate is always “close enough,” until it isn’t.

The failure mode is honest and narrow: if the page renders to canvas, or its markup is a <div> soup with no roles or labels — which is most of the web — the tree is thin or empty and you’re back to pixels. Both Playwright and Chrome DevTools keep take_screenshot alongside take_snapshot for exactly that fallback. The rule is not “never screenshot.” It is “don’t screenshot when the DOM will tell you the answer for a hundredth of the tokens.”

If you take one config change from this piece, it’s this level. Any browser-driving agent you run should be snapshot-first today.

Level 3: page-declared tools

Levels 1 and 2 both make the agent read your page and reverse-engineer what it can do. WebMCP flips it. The page declares its actions as typed tools, and the agent calls a function.

The imperative form, from the spec:

navigator.modelContext.registerTool({
  name: "add_to_cart",
  description: "Add a product to the shopping cart",
  inputSchema: {
    type: "object",
    properties: { sku: { type: "string" }, quantity: { type: "integer" } },
    required: ["sku"]
  },
  async execute({ sku, quantity = 1 }) {
    await cart.add(sku, quantity);
    return { content: [{ type: "text", text: `Added ${quantity}× ${sku}` }] };
  }
});

The agent never walks the DOM. It sees a tool named add_to_cart with a schema, and calls it. The browser translates your registered tools into MCP for whatever agent is attached — Chrome DevTools for agents already enumerates and runs them through a dedicated tool category. Zero pixels, zero refs, a typed call against a function you wrote. This is the cheapest and most reliable rung, and it is the only one that requires you to own the page.

It should feel familiar. Two weeks ago we argued that documentation became an agent distribution channel — that being callable (an MCP endpoint) beat publishing a passive file an agent has to read. WebMCP is that same move one layer in: don’t make the agent read your running UI, make your running UI callable. The button and the tool describe the same action; one is drawn for eyes, the other declared for agents.

The part where I stay honest

Three cautions, because a Builder column that only sells the new thing is marketing.

WebMCP is early. The spec says it plainly: a Draft Community Group Report, “not a W3C Standard,” an origin trial in Chrome and native in Edge, with Mozilla and Apple at the table but nothing shipped cross-browser. Don’t rebuild your UX on it. Prototype two or three tools behind the flag for your highest-value actions and treat the rest as a bet on where this goes.

A page-declared tool is an authenticated action with no brake. add_to_cart runs in the user’s own session, same-origin, with their cookies. The spec gates tools behind a tools Permissions-Policy and secure context — but on the consent question, the human-in-the-loop that should sit between “agent decided” and “action taken,” it punts: no explicit consent mechanism, responsibility delegated to “the agent provider and user agent.” That is the autonomy-before-brakes pattern showing up in a new place. The spec even ships an untrustedContentHint annotation, an admission that a tool’s return value can carry a prompt injection straight back into the agent’s context. Treat every tool result as untrusted input, because the standard already tells you it is.

This could be llms.txt all over again. We’ve watched a “just publish this for the agents” standard land with no agent actually consuming it. WebMCP’s tell that it’s real is not the spec draft or the vendor list — it’s a runtime that actually calls the tools. Chrome DevTools for agents doing so is the first such runtime, and it’s one vendor calling its own standard. The moment that matters is the second, independent one.

Do / watch / ignore

Do. Switch any browser-driving agent from screenshot-first to snapshot-first this week — browser_snapshot in Playwright MCP, take_snapshot in Chrome DevTools for agents. It’s a config change, and it cuts both the token bill and the flake rate at once. Keep take_screenshot wired as the canvas/visual-verify fallback, not the default. If you own a webapp, register two or three WebMCP tools for your most-automated actions and put a real confirmation step in front of any tool that writes.

Watch. One number decides whether Level 3 is a platform or a footnote: does a second agent runtime, not built by the browser vendor, actually call WebMCP tools — and does the consent model get specified before the origin trial graduates? Until then, the accessibility tree is where the durable structure lives.

Ignore. The “agents will browse the web like humans, with eyes” framing. Pixels are the fallback, not the destination — the whole gradient runs the other way, toward more structure and less guessing. And ignore the urge to rewrite your entire UI as tools. Ship three, not thirty.

The through-line across all three levels is one idea: every bit of structure your page gives the agent is a bit the agent doesn’t have to guess, and guessing is what costs tokens and breaks builds. Pixels guess the most. Page-declared tools guess nothing. This week the whole ladder became something you can actually stand on — so stop making the machine squint at a photograph of software it could just be told about.

Prediction: by 2027-Q1, WebMCP (navigator.modelContext) stays an origin-trial / Community-Group draft with no cross-browser-shipped, specified consent model, AND the dominant page-perception path in shipped agent harnesses stays the accessibility-tree snapshot — not page-declared tools, and not vision-first. The structure lives in the a11y tree before it lives in the page’s own tools. Confidence 70%.