Building MacUse: Letting AI Agents Drive a Mac, One Approval at a Time

I kept hitting the same wall with agents. They could edit my code, run my tests, and drive a headless browser, but the moment a task spilled into a real desktop app, they were blind. Fill in this native form, read what that menu bar app is showing, click through a dialog that only exists in a GUI. So I built MacUse: a way for an AI agent to operate the apps on my Mac, with me holding the approval button for every single app it tries to touch.

This is what it does and the handful of decisions that made it something I actually trust to run.

One binary, two roles

MacUse is a single binary that plays two parts.

macuse serve runs on the Mac as a menu bar app. It is the piece that holds the macOS Accessibility and Screen Recording permissions, and it is the only piece that ever talks to an actual app. It serves a small REST API on 127.0.0.1:7710 and, when a request needs your blessing, pops a card under the menu bar icon.

macuse mcp is an MCP server over stdio. It is a thin client that turns an agent's tool calls into HTTP requests against that API. It can sit next to the server on the Mac, or run inside a Linux container and reach back to the host through host.docker.internal.

The whole chain is: agent → macuse mcp over stdio → HTTP with a bearer token → macuse serve → the app. Splitting it this way was the first decision that mattered. Every sensitive permission lives in one place, on the Mac, tied to one signed app. The client that an agent actually speaks to carries no privileges of its own, so it is safe to drop into a sandboxed container alongside the agent.

What the agent can see and do

An agent gets a compact set of tools: list_apps, start_app, get_state, click, type, press_key, scroll, drag, set_value, and batch. get_state is the one that unlocks the rest. It hands back the focused window's accessibility tree, one element per line with a numbered index, and a screenshot when you ask for one. The action tools take those indexes, so the agent looks first and then acts on what it saw. It can also ask for just what changed since its last read, which keeps a busy window from flooding the context.

batch came after the first release. Filling a form one call at a time is a long string of round trips. With batch the agent sends up to a hundred actions against one app in a single call, things like the fields of a form or the strokes of a drawing. Every action is checked before any of them is sent, the run stops at the first one that fails, and an app that needs to be in front is brought forward once for the whole run instead of once per keystroke.

The approval card

The part I care about most is the gate. The first time an agent reaches for an app, the call blocks and a card appears in the popover. It shows who is asking, the client name, the app's bundle ID and its signing team, and exactly what action is about to happen. You get three answers:

• Deny. Refuse this one call.
• This session. Allow the app until the agent's session ends.
• Always. Allow it persistently.

The detail I am proudest of is that Always is keyed to bundle ID plus signing team, not the bundle ID alone. A look-alike build of an app from a different developer has a different signature, so it asks again instead of inheriting trust it was never given. A card nobody answers refuses itself after two minutes by default, so a laptop you walked away from fails closed.

Some apps are never offered at all. Terminals are on that list because controlling one means running arbitrary commands. Automator, Script Editor and Shortcuts are there for the same reason. System Settings is there because it holds the privacy grants. The keychain and password managers are there because they hold your secrets. MacUse itself is there because otherwise an agent could approve its own requests. You can add more apps with deny_apps in the config.

Keeping myself honest

Two more things round it out, both about staying able to answer "what did it just do?"

Every allowed or refused call is appended to ~/.macuse/logs/audit.jsonl. For typing it records the character count, not the characters, so the log never becomes the place your passwords leak. The popover also keeps a running view of recent activity, a per-session Stop, a Pause all, and a list of the apps you have allowed or denied with a Forget next to each.

The API listens on loopback only and every /v1 call needs a bearer token. I am blunt about the caveat in the README: anything that holds that token can ask for app control, and on a Mac running Docker Desktop that includes containers reaching the port. Approvals still gate the access, but the token is a password, so I treat it like one. The popover itself uses a separate key that is regenerated on every launch and never written to disk.

Shipping it like a real Mac app

Distribution turned out to be half the work. The app is signed and notarized with Apple's notarytool, shipped as a Homebrew cask and a DMG, and it updates itself, verifying the SHA-256 and the Developer ID team signature before it replaces its own bits on disk. Because it self-updates, brew upgrade leaves it alone unless you pass --greedy, which is the right default for something that manages its own lifecycle.

Under the hood it is Go, with the same rule I hold everything else to: one hundred percent test coverage, enforced in CI. The macOS-specific accessibility code sits behind a Platform interface, so the portable core, meaning the policy, the approvals, the API and the MCP client, is fully tested without a real desktop in the loop. On a Mac the coverage check runs in Docker. In CI it runs directly.

Try it

brew install --cask radutopala/tap/macuse

Open it, grant Accessibility and Screen Recording from the menu bar, then wire it into Claude Code on the Mac:

claude mcp add macuse -- macuse mcp

In a container, run macuse token on the Mac and hand the Linux client the MACUSE_TOKEN. The source and the full docs live at github.com/radutopala/macuse. It needs macOS 13 or later. If you point an agent at something unexpected with it, I would like to hear what.