feat(desktop): Computer Use for macOS, Windows, and Linux - #6537

…an drive apps

macOS denies every Apple Event with errAEEventNotPermitted (-1743) and never
shows the Automation prompt when an app has no NSAppleEventsUsageDescription.
Codex Computer Use drives other apps through Apple Events, so both the dev
runtime bundle and packaged builds silently failed every desktop automation
regardless of which model or provider requested it.

Declare the key in both places that build a T3 Code bundle: the dev launcher's
Info.plist patch and the electron-builder mac config.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…xisting runtimes

buildMacLauncher early-returns when metadata.json matches, skipping the bundle
copy and every Info.plist patch. Adding NSAppleEventsUsageDescription without
bumping LAUNCHER_VERSION therefore only reached contributors with no
.electron-runtime yet; anyone with an existing dev bundle would keep the old
plist and keep hitting errAEEventNotPermitted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Gives agents computer use on macOS: reading the accessibility tree,
clicking, typing, scrolling and screenshotting — all against a
backgrounded app, so the user keeps working while a task runs.

native/t3-desktop-mcp is an MCP server shipped as a bare executable
rather than an app bundle, so it inherits the host app's TCC grants
instead of needing its own. Background input needs three things
together, and missing any one fails silently: window-addressed events,
field stamping via SLEventSetIntegerValueField (CGEvent's public setter
drops undocumented fields), and activate-without-raise. Ported from
trycua/cua, which takes focus-without-raise from yabai. These are
private SkyLight SPIs resolved by dlsym, so every path falls back to the
global event tap — which works, but moves the user's cursor.

native/t3-chrome-extension covers the browser, where synthetic input
cannot reach: a window only renders its active tab, so coordinate clicks
go blind the moment the user switches away. The extension opens inactive
tabs in a labelled group inside the user's signed-in Chrome and drives
them over the DevTools protocol. It reaches the server through a native
messaging host bridged by a unix socket; a server that finds a live
listener defers rather than stealing the browser from another session.
Tabs and their group are cleaned up when the agent finishes or exits.

An overlay draws the agent's cursor so its actions are visible, both on
the desktop and inside controlled pages.

Registered for the Claude, Grok and Cursor adapters. macOS only —
resolveDesktopMcpPath returns undefined elsewhere and the tools simply
do not appear.

Verified end to end against live apps by probe/verify.py in the
cu-playground checkout: 24 checks covering background input, focus and
cursor non-interference, and the browser flow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Pin the macOS-only NSAppleEventsUsageDescription so a future refactor cannot
drop it and silently break desktop automation again, and assert non-macOS
builds get no mac config at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The bridge came from Nelglor's reference branch (pingdotgg#2156), whose
author explicitly offered it as reference code rather than for merge. It wires
Codex's node_repl transport to a Desktop-owned named pipe, which the native
t3-desktop-mcp server replaces on every platform we target.

Keeps the cross-platform approval plumbing — the part the issue actually asks
for — and removes the thread config override that only existed to feed the
bridge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Mirrors the macOS Swift server's 23-tool surface on UI Automation, so a model
needs no per-platform knowledge: same tool names, arguments and tool text.
Screen capture, display enumeration and app discovery are shared through xcap
because only the accessibility tree and synthetic input genuinely differ.

Verified on Windows 11 (26200) against real applications: list_apps,
list_displays, get_app_state, screenshot, activate_app, click, press_key and
type_text all work end to end.

Linux shares this crate and lands next; the browser tools stay advertised but
report that the Chrome extension host is not wired up off macOS yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

AT-SPI for the accessibility tree, XTEST for synthetic input, sharing the MCP
layer and tool surface with the Windows backend.

Two platform truths shape it, and both are reported to the model rather than
hidden. Wayland clients cannot learn their absolute screen position, so AT-SPI
returns window-relative geometry and coordinate clicks land in the wrong place;
element clicks therefore invoke the accessibility action instead, mirroring the
Windows Invoke pattern. And xcap panics rather than erroring on compositors it
does not recognise, so capture calls are panic-guarded and the release profile
no longer aborts — a screenshot must not take the server down.

Verified on Ubuntu 26.04 under WSLg: list_apps via the a11y bus, get_app_state
returning a real GTK tree with element ids, and click confirmed by driving a
zenity dialog's OK button. Typing lands through XTEST but is intermittent in
this environment specifically — WSLg's XWayland input path is unreliable, and
xdotool fares worse there — so it still wants a check on a plain X11 desktop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Stage the Rust crate for non-macOS artifacts and let the resolver find it, so
the desktop tools are offered on all three platforms instead of macOS alone.
Windows keeps the .exe suffix; the staged directory name does not, which the
resolver now accounts for.

The tools stay unadvertised on platforms with no backend, which the tests pin
so a future platform cannot silently inherit a half-working tool surface.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… keys

Typing was intermittent because XTEST delivers to whatever the compositor
considers focused, which is not reliably the element we were asked to type
into. Addressing the element directly through EditableText removes the race,
and the same interfaces make set_value and select_text work rather than
returning 'not supported'. activate_app now focuses the app's frame, the
closest portable equivalent to raising a window.

Synthetic keys remain the fallback for typing with no target element, and for
press_key chords.

Verified on Ubuntu 26.04: typing, selecting, replacing a value and pressing OK
in a zenity dialog now succeed deterministically, with the dialog returning the
written text.

(--no-verify: the pre-commit formatter has no Rust target and fails on a
Rust-only change.)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Implements the native-messaging bridge the macOS server already had, so the ten
browser_* tools work on all three platforms instead of returning 'not wired up
here'. Chrome spawns this binary as its host, and the host relays to the MCP
server over a local socket — a named pipe on Windows, a Unix socket on Linux —
because Chrome owns the host's lifetime and the server has its own.

The wire messages match the Swift bridge exactly, so the one extension build
serves every platform. Installers register the host the way each OS expects: a
manifest directory on macOS and Linux, a registry key on Windows.

Verified on both platforms by standing in for Chrome: a browser_list_tabs call
travels server → socket → host → extension and renders the tab group back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

vp staged handed every staged path to oxfmt. Given only files it cannot parse
it exits non-zero — 'Expected at least one target file' — so any commit
touching just native/ Rust or Swift sources failed the hook and had to be made
with --no-verify.

Match the glob to what oxfmt actually formats, so a native-only commit skips
the task instead of invoking the formatter with nothing to do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…esent

xcap selects Wayland whenever WAYLAND_DISPLAY is set, and its Wayland path
panics on compositors whose protocol version it does not know, while the X11
path works fine on the same machine through XWayland. Steer capture to X11 for
the duration of the call and restore the variable afterwards.

list_displays now succeeds under WSLg where it previously took the panic guard.
Display capture there still fails, but reports an xcb protocol error rather
than aborting — WSLg's virtual 640x480 rdp-0 root is an unusual X11 target.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

xcap's Wayland path fails to connect on wlroots compositors where the protocol
demonstrably works: under a headless sway session grim captures the same output
fine while xcap reports a display-string parse error. Fall back to grim, which
is the reference wlr-screencopy client, and only advertise the display once a
probe capture has actually succeeded — an advertised display the model then
cannot capture would be worse than none.

GNOME and KDE do not implement wlr-screencopy, so there the existing error
stands and the accessibility tools remain the answer.

Verified on Ubuntu 26.04 against two purpose-built sessions rather than WSLg,
whose virtual rdp-0 root is the odd one out: Xvfb for real X11 and sway
--headless for real Wayland. Both now return a screenshot of the running
dialog; list_displays works on both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

stageDesktopMcpRust builds with --locked, matching the resource monitor, so a
clean checkout without a committed Cargo.lock fails the artifact build outright.
resource-monitor commits its lock for exactly this reason; this crate now does
too.

The lock resolves every target, not just the host's, so one file serves the
Windows and Linux builds. Verified by building both targets with --locked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The .exe suffix was decided inline in both the artifact staging and the
server's resolver. They have to agree exactly — a mismatch means the packaged
app looks for a file that is not there and silently offers no desktop tools —
so give it a name and pin it with a test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Chrome answers browser_open_tab before the tab loads, so the reply carries no
title and no url and the tool rendered 'opened tab_id=692720174 —  []'. A model
reads that as a failed open. Echo the requested address instead, matching what
the macOS server says.

Found by driving the real Chrome extension on Windows; the output now matches
macOS word for word.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

click, type, press, navigate, select_tab and close_tab all fell through to a
bare '<command> ok', which tells a model nothing about what happened or where.
Name the tab and the action, matching the macOS wording so the same feedback
reads the same on either platform.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…er's

Desktop control should never fight the person at the machine for their mouse.
Clicks already preferred a background path that posts straight to a window, but
that left nothing on screen to show where the agent was working, and a
coordinate click without an app name still fell through to the shared cursor.

Draw an agent pointer instead: a small click-through overlay that follows the
action and fades when it stops. AppKit needs a main run loop and the server's
is busy on stdio, so it runs as a child process in cursor-overlay mode.

Coordinate clicks now also resolve the window under the point, so omitting
'app' no longer costs the user their cursor. The CoreGraphics window info
arrives as NSNumber, which does not bridge to pid_t or UInt32 — casting
directly returned nil and the lookup failed silently every time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…und clicks

Add a Settings → Computer Use page that gates t3-desktop MCP injection and
passes agent-cursor / browser-control env into the native servers. Polish the
Chrome extension pointer assets, honor those env flags in Swift/Rust, and
prefer PostMessage clicks on Windows so the user's cursor stays put.

Co-authored-by: Cursor <cursoragent@cursor.com>

Surface permission status and browser extension detection in settings, and
fix ScreenToClient so the Windows MCP builds and background-clicks.

Co-authored-by: Cursor <cursoragent@cursor.com>

Use the repo's Effect/vite-plus test imports and allow sync Node FS reads
for Chrome preference probing from IPC.

Co-authored-by: Cursor <cursoragent@cursor.com>

Rename desktop MCP launch module away from the stuck Effect-check path,
decline unrecognized Codex form elicitations instead of methodNotFound,
detect the Windows native-messaging host, and prime background right-clicks.

Co-authored-by: Cursor <cursoragent@cursor.com>

Match existing quoted Codex config override style and satisfy the Effect
JSON lint rule.

Co-authored-by: Cursor <cursoragent@cursor.com>

Bring Windows Computer Use closer to Mac parity with a layered agent
pointer overlay, and fix Chrome native-host install/reconnect so the
extension can attach on modern Chrome (no BOM manifests, SW wake ping).

Co-authored-by: Cursor <cursoragent@cursor.com>

Keep Computer Use Rust MCP staging alongside main's bundle
self-containment helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>