Computer use
Sandboxed desktops an agent drives through the accessibility tree and a person can watch, take over and hand back.
A Computer is an isolated Linux desktop session that an AI agent drives. A person can watch it live, take over the mouse and keyboard, and hand it back. At every moment exactly one party may inject input, every change of hands is fenced and audited, and nothing a viewer sends reaches the desktop unless the arbiter allows it.
How the agent sees and acts
Perception comes from the AT-SPI2 accessibility bus, not screenshots. Actions go through accessibility interfaces first and through /dev/uinput only when no interface can carry them, so it works on Wayland. Every action runs a guard chain: identity, offer, scope, live re-identification, disabled check, safety gate, side effect, verification, audit.
Three faces, one contract
| Face | Binary | Transport |
|---|---|---|
| MCP server | atspi-kbm-mcp |
JSON-RPC 2.0 over stdio |
| CUA adapter | cua-adapter |
one JSON request per line on stdin and stdout |
| Ego-lite | atspi-ego-lite |
line-delimited JSON on a loopback Unix socket, token-gated, with rate and time budgets |
The gateway
The gateway is an HTTP and WebSocket service that creates computers, streams them to a browser as RFB over WebSocket (noVNC compatible), arbitrates takeover and serves the audit log.
cargo build --release -p computer-gateway
computer-gateway --print-default-config > gateway.json
computer-gateway --config gateway.json| Method and path | Does |
|---|---|
POST /v1/computers |
Create a computer |
GET /v1/computers/{id} |
Status |
DELETE /v1/computers/{id} |
Destroy |
POST /v1/computers/{id}/tokens, GET |
Mint and list capability tokens |
DELETE /v1/computers/{id}/tokens/{token_id} |
Revoke a token |
POST /v1/computers/{id}/takeover |
A person takes input |
POST /v1/computers/{id}/handback |
Hand input back to the agent |
POST /v1/computers/{id}/agent/resume, GET /agent/guard |
Resume the agent, read its guard state |
POST /v1/computers/{id}/suspend, /unsuspend, /revoke |
Suspend, resume, revoke |
GET /v1/computers/{id}/audit |
The hash-chained audit log |
GET /v1/computers/{id}/stream |
Live desktop over WebSocket |
GET /v1/computers/{id}/tree |
Mobile UI tree |
GET /v1/computers/{id}/display, /settings, /viewers |
Display, settings and viewers |
POST /v1/exec, GET /v1/exec/{id}/events, POST /v1/exec/{id}/cancel |
Run and observe a command in a sandbox |
GET /healthz, GET /metrics |
Health and metrics |
Requests are authenticated by a platform proxy that sets a tenant header; the gateway checks the peer, rate limit, tenant, origin and token on every request.
Isolation tiers
| Tier | Backend |
|---|---|
| Process | An Xvfb desktop as a hardened child process: Landlock, seccomp and cgroups |
| Container | An OCI runtime (runc, crun or youki) with namespaces, cgroup limits, a seccomp baseline and deny-by-default network egress |
| Micro-VM | Firecracker, cloud-hypervisor or Kata, with a host-guest vsock protocol |
The Android backend gives mobile computers a UI tree through adb and scrcpy.
Audit
Every change of hands is recorded in a SHA-256 hash-chained log, with durable append-only storage, signed external head anchors and a verifier. A tail truncation is only detectable against an externally anchored head.
Hosted API
The Webfold API can front a gateway for your account. POST /v1/computers creates a computer, and the /v1/computers/{id} endpoints cover status, destroy, capability tokens, agent resume and guard, takeover, handback, revoke, suspend, unsuspend and the audit log. The tenant is taken from your API key, never from the request, and a gateway capability token goes in the X-Computer-Token header where a call needs one. The live desktop is not proxied: GET /v1/computers/{id}/stream returns the gateway address, and you connect to it with a view or control token. The endpoints answer 503 until the API has a gateway configured. See the API reference.