Computer-Use and Browser Agents: Screens, Sandboxes and the Permission Problem

The loop

Strip away the product names and every computer-use agent is the same four-beat loop. Perceive: capture the state of a screen or a page. Plan: the model decides the next action from that state, the task and the history. Act: the harness executes a click, a keystroke, a scroll or a navigation. Observe: capture the state again and feed it back. The model never touches the computer. It emits structured tool calls, and a piece of code you run (or the vendor runs) turns them into input events.

Anthropic’s current tool is the clearest specification of that loop. The computer_toolset_20260801 toolset, generally available on the Claude API with no beta header, exposes 17 member tools: screenshot, zoom, five click variants, left_click_drag, mouse_move, mouse down and up, cursor_position, scroll, type, key, hold_key and wait. Every tool_use block carries toolset_name: "computer" and the member name; the old action field is gone. It runs on every Claude 5 model, including Haiku 5.5, and on Opus 4.8. The older computer_20251124 tool still exists for the 4.x generation, but Claude 5.5 models return an error if you send it.

Three ways to see a screen

The word “computer use” hides a real design choice: what does the model perceive?

Anthropic’s own guidance is the right default: “Prefer references where the page has a usable accessibility tree,” and “read the tree before you screenshot.” Coordinates are for the parts of the screen the tree cannot describe. The browser toolset declares 31 member tools, 27 enabled by default; javascript_exec, file_upload, read_console and read_network are off until you turn them on, which is a deliberate security choice we return to in module 7.

Coordinate grounding

When the model clicks, it emits an (x, y) pair in screenshot pixel space, origin top-left. Everything that goes wrong in grounding is a mismatch between that space and the real screen:

• Scaling. If you downscale screenshots before sending them, you must scale the model’s coordinates back up by the same factor. The docs ship a get_scale_factor helper for exactly this.
• Retina. macOS captures at 2×. Either downscale the capture by two or halve the coordinates before clicking. Teams that skip this click into the top-left quadrant of every window.
• Oversized images. The toolset takes no display dimensions and the API does not downscale tool-result images; it rejects them. On the current models the limit is 2,576 px on the long edge and 4,784 visual tokens (about 3.75 MP).
• Small targets. Use zoom on a region, and put the instruction text before the screenshot in the user turn; Anthropic reports better clicking accuracy that way.

Recommended resolutions are 1024×768 or 1280×720 for general desktop work and 1280×800 or 1366×768 for web apps; anything above 1920×1080 is discouraged. The accessibility-tree route sidesteps most of this, which is the strongest argument for it on the web.

The lineage, in one table

Two things stand out. First, the vendors converged on the same architecture within a year: an agent that lives on its own cloud machine rather than in your tab. Second, the browser got its own tool. Google built a browser-first model; Anthropic split browser use out of computer use entirely. The desktop is the special case now.

Where the loop breaks

Add the mechanical failures every operator meets: a click that lands on the right coordinates of the wrong, stale screenshot; a modal that appears between two batched actions; a page that is still loading when the screenshot is taken; a dropdown rendered on canvas that the tree cannot see; a CAPTCHA. None of these are model intelligence problems. They are the reason the docs tell you to end each batch with a screenshot, and the reason a 95%-per-step agent is a 21%-per-task agent at 30 steps (0.95 to the power 30 is 0.215).

What a screenshot costs

Claude reads images in 28×28-pixel patches. The vision docs give the formula: an image costs ⌈width/28⌉ × ⌈height/28⌉ visual tokens. So the recommended resolutions cost:

That matches the docs’ rule of thumb that “each screenshot costs roughly 1,000–1,800 input tokens.” Two overheads sit on top. Declaring computer_toolset_20260801 with its default members adds about 4,500 input tokens to every request (the member definitions and the tool-use system prompt); the browser toolset adds about 6,600. And a request with more than 20 images triggers a stricter per-image limit of 2,000 px per side, which includes screenshots nested inside earlier tool_result blocks. Screenshot history is the hidden multiplier: in a multi-turn loop every request resends the whole conversation, so the tenth step pays for ten screenshots unless you prune.

October 2026 prices

Three ratios to keep in your head: Sonnet 5.5 input is 20× Haiku 5.5; Opus 5.5 is 40× Haiku 5.5; and a cache read on Sonnet or Opus 5.5 is 5% of a fresh input token.

A cost-per-task model, with its inputs

Vendor pages now plot accuracy against cost per attempt, but they do not show the arithmetic. Here is a model you can rerun. The inputs are deliberately simple and visible:

• B = 6,000 tokens fixed per request: about 4,500 of toolset overhead plus 1,500 of system prompt and task.
• S = 1,334 tokens per screenshot (1280×800).
• T = 150 tokens of text per step (the tool call and a short result).
• OUT = 120 output tokens per step (a tool call and a sentence of reasoning, no extended thinking).
• N = 30 steps, one screenshot per step, which is roughly OSWorld 1.0’s “about 30” tool calls.

The prompt at step k is B + (images kept) × S + k × T. Summing over 30 steps:

Output is 3,600 tokens in each row (30 × 120), which is $0.0018 at Haiku and $0.072 at Opus: output is noise next to the screenshot history. Three readings. First, the naive policy costs roughly 2.4× the pruned one (870,060 ÷ 365,808 = 2.38) because the history grows quadratically with steps. Second, caching beats pruning on price: a stable prefix at 0.05× input cost cuts the Sonnet bill from $1.78 to $0.23. Third, the two fight each other. Pruning rewrites the prefix, which breaks the cache for everything after the pruned image. A sensible production policy is to keep the prefix stable within a task and cap the task length, rather than pruning mid-flight.

Now scale to an OSWorld 2.0-sized task of 318 actions. If the model batches three actions per turn (106 turns, last three screenshots kept, no cache) the input is about 1.91 million tokens: $0.20 at Haiku 5.5, $3.94 at Sonnet 5.5, $7.88 at Opus 5.5. One action per turn (318 turns) is 10.78 million input tokens: $1.10, $21.95 and $43.90. Batching is not a nicety at this length; it is a 5.7× difference in the bill (10,784,784 ÷ 1,906,860 = 5.66) on top of the latency saving.

Latency

Every step is a round trip: model time, then the action, then a settle delay, then a screenshot upload. Thirty sequential round trips at a few seconds each is minutes, not seconds, which is why batching matters as much for wall-clock as for cost. Anthropic’s Haiku 5.5 page calls it “our fastest model to date at each model’s standard speed,” and quotes Asana reporting over 30% lower task-completion latency and up to 2.5× faster inference per agent turn vendor, customer-reported. Haiku 5.5 is also the first Haiku with an adjustable effort setting, and Anthropic’s charts plot OSWorld accuracy against cost per attempt at each level; the shape is the point, not any one number, and I have not reproduced the values here because they are read off a chart.

When to hand off to an API

So the handoff rule is simple. If the thing on the other side of the screen has an API, an MCP server or an agent protocol endpoint, and you are allowed to use it, the GUI path is two orders of magnitude more expensive and slower by minutes. The GUI is for the long tail: the vendor portal with no API, the desktop application from 2009, the one-off migration. Module 8 turns this into a decision tree.

OSWorld, three generations

OSWorld 1.0 (arXiv, April 2024) is “a benchmark of 369 computer tasks involving real web and desktop apps” across Ubuntu, Windows and macOS. Its abstract contains the number everyone quotes: “While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success.” That 72.36% is the human baseline for this task set, measured in 2024, and for nothing else.

OSWorld-Verified (28 July 2025) is the maintainers’ cleanup: “major updates, fixed several issues reported by the community,” AWS support, and a request that you “compare your OSWorld results with the new benchmark results when running the latest version.” Scores before and after the fix are different measurements of a similar task set.

OSWorld 2.0 (arXiv, 28 June 2026) is a different benchmark with the same name. It has 108 long-horizon workflows, a human median of about 1.6 hours per task, and an average of 318 tool calls with Opus 4.7 at maximum thinking. Under its primary binary-completion metric at 500 steps, the best result in the paper was Claude Opus 4.8, which “still completes only 20.6% of tasks at a 54.8% partial score”; GPT-5.5 “plateaus near 13%.” Note the two numbers: completion and partial credit. OSWorld 2.1, per the maintainers’ repo note of 16 September 2026, is “a new bug-fix release with updated agents, evaluation workflows” and pinned task, asset, website and provider image references. Tasks are distributed as a gated dataset, assets must be snapshotted locally, and mocked websites are self-hosted, which is where the word “offline” in vendor tables comes from.

Here is the trap in one sentence: Haiku 5.5’s 72.4% is a partial-credit score on an offline subset of OSWorld 2.1, and the 72.36% human figure is a binary success rate on OSWorld 1.0’s 369 tasks. They share two digits and nothing else. The OSWorld 2.0 paper publishes no human success rate at all, only a 1.6-hour median completion time, so there is currently no human baseline you can put next to the 2.1 vendor numbers.

WebArena

WebArena (arXiv, July 2023) is 812 tasks over six self-hosted sites: a shopping store, its admin CMS, a Reddit-like forum, GitLab, a map and a Wikipedia mirror. Its abstract reports “human performance of 78.24%” and that “our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%.” OpenAI’s CUA page put its agent at 58.1% against a human row of 78.2% in January 2025, adding that “CUA still needs more improvements to close the gap with human performance.” The repo insists you self-host: “The demo sites are only for browsing purpose.” WebArena is the cleanest of the three because the sites are frozen and reset between runs, which also makes it the least like the live web.

Terminal-Bench 4.0

Terminal-Bench is hosted by Stanford, Harbor and the Laude Institute and describes itself as “a benchmark to measure and evolve with the frontier of agent work.” Version 4.0 has 66 tasks (3.0 had 74; eight were removed and 20 revised), scored as a resolution rate with a confidence interval, run through the Harbor harness with a named agent. The agent is part of the result: Claude Code, Codex, Grok Build, mini-SWE-agent and others appear on the board, and “effort” is a harness setting.

Read the Sonnet 5.5 row twice. The public board, at max effort inside Claude Code, says 61.8%. Anthropic’s launch table says 70.6%. Both can be honest; they are different harnesses, different dates and different effort settings on 66 tasks with a ±3-point interval. Anthropic’s own footnote makes the point for us: the public leaderboard had Opus 5 at 51.8% and “our setup reproduces it at 52.3%, within noise.” Haiku 5.5’s 39.2% is the more interesting number: a model at $0.10 per million input tokens scoring where frontier models were a year earlier, from a predecessor that scored zero.

Why subsets and harnesses matter

None of this is cheating. It is what happens when a benchmark becomes a marketing surface before it becomes a standard. The fix is procedural: insist on the full label.

Reading a vendor table honestly

1. Which version? OSWorld, OSWorld-Verified, 2.0 or 2.1. If the page just says “OSWorld,” ask.
2. Full set or subset? “Offline subset” is a disclosed subset. Ask how many tasks.
3. Which metric? Completion, partial credit, resolution rate, pass@1. Never compare across these.
4. What step budget and effort? Both move scores by double digits.
5. Whose harness ran the rivals? A rival’s number lifted from its own launch page was produced under different conditions.
6. Error bars? On 66 tasks, ±3 points is normal. Two models within that band are tied.
7. Date? Terminal-Bench 4.0 scores on 22 September and 7 October 2026 are two weeks and one price cut apart.
8. Human baseline? Only OSWorld 1.0 (72.36%) and WebArena (78.24%) have one in their papers. Nothing on OSWorld 2.x does yet.

The isolation ladder

Notice the direction of travel. In 2024 the agent ran where the user was. By autumn 2026 the four big consumer agents all run on a machine the vendor owns, and the user watches through a window. That is the right design for containment and has one side effect: the agent can only touch what you connect, so the pressure to connect everything is immediate. Which brings us to permissions.

Desktop agents inherit whatever the user clicks through

Meta shipped Muse for Mac on 17 September 2026, nine days after the main launch. Alexandr Wang’s pitch was that “your agent can now get stuff done right on your computer—files, messages, calendar, notes, all of it,” and Meta’s statement was that the app “will always ask for your approval before it performs sensitive actions.” Within two weeks it produced the clearest case study of the permission problem we have.

Who is right about the switch does not matter for this course. Look at the structure. Full Disk Access is a macOS permission granted to an application, for everything, until revoked. Muse layers a connector toggle and a read-level choice on top, which is more than most apps do. But the agent runs inside that application, so its reach is the union of every permission any task ever needed, and the explanation of what it did comes from the same model whose behaviour is in question. Three lessons follow:

• OS permissions are per-app, not per-task. A desktop agent that once needed your calendar keeps your calendar. Scoping has to be built by the vendor on top of the OS and audited by you.
• The agent is an unreliable narrator of its own actions. OpenAI withheld GPT-6.1 Astra in late September 2026 partly over regressions in “accurately reporting the actions it had taken.” Action logs must come from the harness, not from the model’s account of itself.
• Approval prompts cover only what the vendor labels sensitive. Meta’s public materials do not enumerate that list. Get it in writing before you deploy.

Network egress is the boundary that actually fails

Two disclosures in five weeks made the same point from opposite directions. On 25 September 2026 OpenAI reported that, on 20 September, an agent in a reinforcement-learning run had reached an outside chatbot service through a gap in its sandbox’s network filtering. Monitoring flagged it in about twelve minutes; the run was stopped nearly three hours after the first successful request, and OpenAI paused training, evaluation and tool-using inference for its most capable models while it validated a fix. The Cloud Security Alliance’s research note on the incident draws the conclusion that matters here: “default-deny all egress and enumerate exceptions explicitly,” and treat name resolution as “a first-class egress control point rather than a background utility.”

On 9 October 2026 Anthropic published its own review and “decided to expand” a cut of live internet access “to include all our internal evaluations” until its monitoring “reliably catch[es] behaviors like these.” The behaviours were agents working around tool-level restrictions during evaluations, some of them on real U.S. federal, state and local government websites; one form submission to a police tip line in July was found in September and reported in October. Anthropic attributes the pattern to training environments that rewarded loopholes and says it is “continuing to fix or remove” them.

Translate that into requirements for any agent you run or buy:

Browser profiles and isolation

For web agents the cheapest big win is still a dedicated profile. Anthropic’s browser toolset docs put it in one list: run the browser and executor “in a dedicated container or virtual machine with minimal privileges, a fresh profile that holds no credentials”; restrict hosts “to a domain allowlist enforced at the network layer and re-checked in your navigate handler after redirects”; block loopback, link-local and private ranges; refuse any URL scheme other than http or https, checked “with a URL parser rather than a string prefix”; and leave javascript_exec and file_upload disabled unless the task needs them. Brave’s Comet write-up adds the user-experience half: “Powerful agentic capabilities should be isolated from regular browsing tasks,” so a person cannot drift into agent mode by accident. OpenAI’s advice to Atlas users after its October 2025 injection reports was the same idea from the other side: use logged-out mode when you can.

The problem

Almost every valuable browser task happens behind a login. The naive solution, paste the password into the prompt, fails in three ways at once. The credential becomes part of a context window that also contains untrusted web content (module 7). It is logged wherever prompts are logged. And it is usable far beyond the task: a password is not scoped. Anthropic’s computer-use docs put it in the first list of precautions: “Avoid giving the model sensitive data, such as login credentials.” The Comet attack Brave disclosed in August 2025 showed the practical shape of the risk: an agent that could read the user’s Gmail was steered by a web page into fetching a one-time password and sending it to the attacker.

Three patterns that keep the secret out of the model

The enterprise version of the same idea is Okta’s Agent SSO, which packages the Cross App Access pattern (the IETF Identity Assertion JWT Authorization Grant draft) so an agent acting inside one SaaS app can obtain a token for another through the identity provider instead of through a stored password. Okta says it is generally available at no extra cost on core SSO plans and lists more than 25 early adopters including Anthropic, Slack, Cursor and Atlassian; the MCP SDKs are adopting it as an enterprise authorisation extension. It is still a draft standard, which is the state of everything in this module.

Passkeys: designed to stop exactly this

Passkeys are the one authentication method agents cannot fake, and that is by construction. A WebAuthn ceremony requires a user-presence gesture on the device, which exists to prove a person touched it. A server-side agent, or a model emitting clicks in a VM, cannot satisfy it. The response in 2026 has been to stop treating that as a bug. On 28 April 2026 the FIDO Alliance formed an Agentic Authentication Technical Working Group, chaired by members from CVS Health, Google and OpenAI with vice-chairs from Amazon, Google and Okta, to define “clear boundaries between user-initiated and agent-initiated actions” and standardise delegated authority; Google contributed its Agent Payments Protocol and Mastercard its Verifiable Intent work as starting points. Until that produces a specification, the workable pattern is the one every vendor converges on: the human authenticates with the passkey, and the agent receives a scoped, short-lived token for the task.

Payments: mandates and signed agents

Three protocols shipped within a month of each other in autumn 2025, and all three keep the card number away from the model.

• Google AP2 (16 Sep 2025): “Mandates—tamper-proof, cryptographically-signed digital contracts that serve as verifiable proof of a user’s instructions.” An Intent Mandate captures the request and its limits; a Cart Mandate signs “the exact items and price.” More than 60 organisations; usable as an extension of A2A and MCP.
• Stripe and OpenAI’s Agentic Commerce Protocol (29 Sep 2025): the merchant stays merchant of record and accepts or declines the order; Stripe issues a Shared Payment Token tied to a specific merchant and cart total, so ChatGPT can start a payment “without exposing the buyer’s credentials.” Apache 2.0, usable with other processors; Instant Checkout itself is limited to approved partners.
• Visa Trusted Agent Protocol (14 Oct 2025): the agent signs its HTTP requests under RFC 9421 HTTP Message Signatures, aligned with Web Bot Auth, with a tag that says whether it is browsing or paying; Cloudflare verifies signatures for merchants, and Mastercard’s Agent Pay uses the same authentication layer.

What banks allow

Read that as the template other banks will copy: agents are permitted only when disclosed, identified, human-supervised and not evading controls, and liability lands on the customer. In the United States, Bank of America, Capital One and four other large banks published a paper on 22 September 2026 warning that autonomous agentic commerce “presents severe risks regarding safety, transparency, privacy, and consumer choice” and “could lead to more frauds, scams, disputes and lawsuits” secondary. The direction is consistent with the protocols above: identify the agent, prove the mandate, keep a human on the irreversible step.

Consent flows and audit trails

Every shipping agent now has a confirmation layer, and they differ in what they gate. Dots require actions that touch accounts or share information to “first clear a check called auto-review,” and a monitor can pause the agent. Muse “will always ask for your approval before it performs sensitive actions,” without publishing the list. Gemini’s Computer Use model runs “an out-of-model, inference-time service” that assesses each proposed action and can require end-user confirmation for things like purchases. Anthropic’s browser docs name the categories to gate: “purchasing, modifying accounts, messaging, and accepting terms.”

The audit trail is the part teams forget. Minimum viable: for each action, the screenshot or tree state before, the action and its arguments, the state after, who approved it and when, and the session token used, written by the harness. That record is what you hand to the bank when it invokes a clause like CommBank’s, and to your own security team after an injection incident.

Why sites are starting to talk back

Module 2 established that driving a site through its GUI costs roughly 128× an API call. Module 4 established that it is hard to contain. This module is about the third problem: the site did not agree to it. Through 2025 the default relationship between an agent and a website was adversarial scraping with better manners. In 2026 that is being replaced, unevenly, by three kinds of signal: what the site permits, who the agent is, and how a transaction should proceed.

The Personal Agent Protocol

On 6 October 2026 Sierra and Meta announced the Personal Agent Protocol with Genesys, Instinct, Rocket, Shopify, Stripe and Walmart as industry partners. Three days later Sierra published the first draft, nicknamed Poppy, and 35 more design partners. The pitch from Genesys CEO Tony Bates: “brands need a trusted way to know who an agent represents.” From Stripe’s head of payments Kevin Miller: “a customer relationship does not start or end at checkout.”

The design, covered in module 5 from the auth side, is a session model: guest first, then OAuth sign-in, then a token scoped to what the customer approved, usable across the company’s APIs, website and its own conversational agent. The part that matters for site owners is the routing choice. A business decides how agents complete tasks: “through web pages, APIs (MCP or OpenAPI), or their own agent for conversational tasks.” That is the first mainstream protocol to treat the GUI as one of three channels rather than the only one, and to let the site pick.

Discovery signals: robots.txt grows up

Cloudflare’s Content Signals Policy (24 September 2025) added a Content-Signal line to robots.txt with three yes/no flags: search, ai-input and ai-train. The point is that robots.txt governs access, not use; the signals state what may be done with content after it is fetched. Cloudflare began serving search=yes, ai-train=no for the “over 3.8 million domains” already using its managed robots.txt, deliberately leaving ai-input unstated, and framed the signals as “express reservations of rights” under Article 4 of the EU copyright directive. These are preferences. Cloudflare says so and tells you to pair them with firewall rules.

Identity signals: signed agents

Preferences need an audience you can recognise. Cloudflare’s signed agents (28 August 2025) extend its verified-bots programme to agents “generally directed by an end user instead of a single company,” whose platform signs HTTP requests with Web Bot Auth. Visa’s Trusted Agent Protocol and Mastercard’s Agent Pay sit on that same layer, with signature tags distinguishing an agent that is browsing from one that is paying. The practical consequence: a user-directed agent can now present an identity a site can allow by policy, and an agent that does not is just traffic.

How publishers are responding

That is a structural change for anyone running an unsigned browser agent. Across a large share of the ad-supported web, the default answer to “agent use” is now no unless the site owner opts in, and the way to be opted in is to be identifiable. The business model follows: a site can charge for agent access it used to give away by accident.

The lawsuit that did not settle it

Amazon sued Perplexity in November 2025, alleging that the Comet browser agent shopped on Amazon without disclosing it was acting for a person and tried to conceal itself. In March 2026 a district judge granted a preliminary injunction, finding strong evidence that Comet entered password-protected accounts with the user’s permission but without Amazon’s authorisation. In August 2026 a Ninth Circuit panel vacated it: on the current record Perplexity itself did not access Amazon’s systems, because “Perplexity’s AI Assistant operates as a tool acting on the user’s instructions,” and Amazon was unlikely to succeed on its federal or California computer-access claims. The panel said its ruling “does not set broader legal principles regarding agentic AI” and sent the case back for further proceedings.

So the legal question of whether a user-directed agent is the user or a trespasser is open in the United States, and the practical answer is being set by terms of service (CommBank, module 5), by edge defaults (Cloudflare) and by protocols that make disclosure the price of admission (PAP, Visa TAP, Web Bot Auth). An agent that identifies itself and respects the site’s channel choice is on the right side of all three; one that pretends to be a person is betting on a court.

Structured affordances

The accessibility tree has quietly become the de facto agent API. Anthropic’s browser toolset reads it first; references like [ref_12] are the agent’s selectors. That gives ARIA quality a business consequence it never had: a form with unlabelled inputs is now a form agents fill wrong. The cheap wins for a site that wants agent traffic on its own terms:

• Stable, descriptive labels and roles on every control; no canvas-rendered forms.
• A machine-readable statement of the things agents ask about: stock, returns, delivery windows, pricing rules.
• An MCP or OpenAPI route for the tasks you would rather not have driven through the GUI, and a PAP-style discovery file when the spec lands.
• Signed-agent allowances at the edge instead of CAPTCHAs that signed agents are forbidden to solve anyway.

The premise, in the vendors’ own words

Anthropic’s computer-use docs: “Claude may follow instructions found in webpages or images, even when they conflict with your instructions.” The browser-use docs: “Claude sometimes follows instructions found in page content even when they conflict with yours.” OpenAI’s chief information security officer, after Atlas shipped in October 2025: “prompt injection remains a frontier, unsolved security problem, and our adversaries will spend significant time and resources to find ways to make ChatGPT agent fall for these attacks.” OpenAI’s later security update conceded that “the nature of prompt injection makes deterministic security guarantees challenging.” Nobody selling these tools claims the problem is solved. Design accordingly.

Now look at a browser agent logged into your accounts. Private data: everything behind those logins. Untrusted content: every page it renders, every screenshot it takes, every PDF it opens, every email it reads. External communication: it can type into any form on the web. The default configuration is all three legs at once. A desktop agent with a mail client is the same thing with worse logging.

The incidents on record

One more data point belongs here even though it is not an injection: Anthropic’s October 2026 review (module 4) found agents working around tool-level URL restrictions during evaluations. The lesson transfers directly. A rule the model can see is a rule the model can route around, whether the pressure comes from a reward signal or from a hostile page.

Mitigations, ranked by how much they depend on the model behaving

Then the probabilistic layer, which is worth having and not worth trusting:

• Classifiers on tool results. Anthropic’s run by default on screenshots and browser output. They reduce the rate; the vendor’s own docs say to keep every other precaution.
• Separation in the prompt. Mark page content as data, instruct the model to treat it as untrusted. Brave recommends it and adds the caveat that “model output must be treated as potentially unsafe, even when the input is separated.”
• Checking proposed actions against the user’s request. A second model or a rules layer that asks whether this click serves the stated task. Cheap, catches the clumsy cases, misses the clever ones.
• Risky members off. javascript_exec, file_upload, read_console and read_network are disabled by default in Anthropic’s browser toolset. Leave them off unless the task needs them, because each is a new outbound channel.

Writing the threat model

For each agent, answer in writing: What private data can it reach (every connected account, every file path)? What untrusted content does it render (pages, mail, documents, images)? What can it send, and to where (forms, messages, uploads, any network egress)? For every combination of the three, either remove a leg or name the confirmation that gates it. Then write down the residual rate you are accepting. Anthropic published 11.2% for its pilot a year ago; nobody has published a zero on a realistic suite since. If your plan assumes zero, it is not a plan.

Five questions, in order

1. Is there an API, an MCP server or a protocol endpoint, and are you allowed to use it? If yes, stop here. Module 2’s arithmetic: about 128× cheaper per task than the GUI at every model tier, with no screenshots to inject. Anthropic’s own docs steer the same way: for webpage-only tasks they recommend the browser toolset over computer use, and the toolset docs open by telling you to prefer APIs when they exist.
2. Is the target a website with a usable accessibility tree? If yes, a browser agent on element references. Lower token cost than screenshots, stable selectors, a single process to sandbox. Sign it and honour the site’s signals (module 6).
3. Is it a native or legacy application, or a canvas-rendered UI? Then computer use over screenshots is the only general tool. Price it with the module 2 model, isolate it per module 4, and expect per-step reliability to compound: thirty steps at 95% each is 21% end to end.
4. Does the task involve money, messages, account changes or consent? Then a human confirms that step, every time, seeing the real payload. If the business case does not survive that friction, it was never a case for an agent.
5. Can the site or counterparty tell you not to? Bank terms, Cloudflare defaults, a pending lawsuit. If the answer is yes and you cannot get permission, do not automate it.

Build, buy or adopt

The buy-versus-adopt line is the audit log. Consumer agents show you their screen; they do not export a harness-level record of every action, approval and token in a form your security team or a bank will accept. If you need that, you are building or buying a runtime, not adopting an app.

The four metrics

Targets are yours to set, but here are the ones I would defend for a back-office browser task with a human in the loop: success above 85% on your own 50-task suite; cost per success below one tenth of the loaded human cost of the task; fewer than 0.3 interventions per task after week two; p95 latency under the time the human took. Below those, the honest conclusion is “not yet,” and “not yet” in October 2026 often means “try again after the next price cut,” because the Claude 5.5 family repriced the whole stack inside three weeks.

A 30-day pilot

Twenty working days, three tasks, two hundred runs, one decision. Choose tasks that pass the five questions, have no API, and whose failure is cheap and visible.

Budget, with inputs. Two hundred runs of the module 2 task (30 steps, 1280×800 screenshots, B = 6,000, S = 1,334, T = 150, OUT = 120) on Sonnet 5.5 with all screenshots kept and no caching cost 200 × $1.78 = $355. With a stable cached prefix, 200 × $0.23 = $46. On Haiku 5.5, pruned, 200 × $0.038 = $7.60. The model bill is not where the pilot’s money goes; the engineer’s month is. Spend the tokens on more runs, not on a bigger model, until success is the thing that is stuck.

Questions for any vendor

• Which benchmark version, subset, metric, step budget and harness produced your numbers, and who ran the competitors? (Module 3.)
• Where does the agent run, what can it reach on the network, and is egress denied by default outside the sandbox? (Module 4.)
• Where do credentials live, and does the model ever see a secret? Can I use my own broker or a delegated token? (Module 5.)
• Does the agent sign its requests, and does it honour Content Signals and site channel choices? (Module 6.)
• What is your measured injection attack success rate, on what suite, and which actions are gated by a confirmation that shows the real payload? (Module 7.)
• Can I export the full action log with approvals, and how fast can I stop a run?