Anthropic introduces computer use with Claude

Announcement record

Anthropic announced computer use in beta alongside updated Claude 3.5 Sonnet and Claude 3.5 Haiku. The feature lets models interpret screenshots and interact with software through tool actions. The announcement described safeguards and limitations for the early release rather than an unrestricted automation capability in practice.

Why it matters

Computer control needs stronger execution boundaries than chat. Use isolated environments, allowlisted actions, approvals for consequential steps, and replayable logs before automating browser or desktop tasks.

What the announcement established

Anthropic's October 22, 2024 post introduced computer use as a public beta API capability alongside the updated Claude 3.5 Sonnet and Claude 3.5 Haiku. It described screenshot interpretation, pointer movement, clicking, and typing, while calling the capability experimental, cumbersome, and error-prone for some ordinary interactions such as scrolling, dragging, and zooming. The original post named low-risk exploration and discussed safety classifiers. Those statements describe that launch state. They do not establish present-day model behavior, safety classification, feature access, or suitability for a particular browser, account, or operating system.

Screen pixels are untrusted input

A computer-use agent sees rendered state, not a signed business contract. Page content, overlays, clipboard fragments, tooltips, and URL text can contain instructions intended to steer the agent away from the user's task. Treat every screen interpretation as a proposal that needs an authenticated target, a validated action schema, and a policy decision outside the model. Hypler's architecture can use a narrow action broker: it receives an intent such as open_ticket_draft, checks role and workspace, then exposes only the allowed navigation or form primitive. The model should never receive ambient credentials or a broad shell simply because it can operate a cursor.

Control failures differ from chat failures

A wrong answer is visible; a wrong click may submit data, change an account, or leave a browser at an unsafe destination. Other common failures include focus landing in the wrong field, stale screenshots after navigation, retries duplicating submissions, and a user taking control while the agent still has queued input. Build a state machine that invalidates pending actions on page change, uses idempotency keys for supported mutations, and pauses after any sensitive boundary. Keep screenshots as limited evidence with redaction policy, not as unrestricted application logs. A replay should show the intended action, observed target, authorization decision, and result.

Hypothetical browser repair

Hypothetical: an operations user asks an agent to prepare a damaged customer profile for review. The agent works in a disposable browser profile with a read-only account token. It identifies two possible records, so it creates neither update and presents both IDs plus the evidence used to distinguish them. After the user chooses one, the agent can populate a draft patch in a local review form. A separate service validates field-level permissions and requires an operator approval before the platform sends the mutation. If an unexpected login modal appears, the run stops with credential_boundary_detected; it does not ask the model to work around the prompt or capture credentials from the screen.

Acceptance checks before use

Test prompt injection rendered inside page text, overlays that move between screenshot and click, stale DOM or screenshot state, a blocked action, a network retry, a duplicate submit, and manual takeover. Assert that the agent cannot navigate outside an allowlisted origin, access a new credential, or perform a mutation without a signed approval. Verify that screenshots redact protected fields before persistence and that retention deletion reaches visual logs. Exercise the stop path after a modal, a selector mismatch, and a policy denial. Success means the system stops safely and produces a useful review artifact; it does not mean the agent completed every task autonomously.

Hypler connection

Hypler applies computer interaction within supervised workflows where the request, policy decision, authorized action, and receipt remain distinct. This pattern lets teams evaluate automation against clear boundaries, inspect a stopped run, and retain human ownership of the result rather than treating cursor control as authority by itself.

Review evidence to retain

A desktop or browser automation review should retain enough evidence to explain the target and outcome without becoming a surveillance archive. Capture the task request, policy grant, redacted before-and-after state, normalized action sequence, final service response, and stop reason. Do not retain unrestricted screen recordings by default. The evidence design should support a user dispute: was the requested record selected, did the page change before click, and did a protected boundary correctly stop the run? Sample completed and stopped runs during evaluation. The goal is not to prove that perception is perfect; it is to make uncertainty and intervention visible to the people responsible for the workflow.

Related capabilities

• Custom software and product engineering