Everyone Just Shipped an AI That Watches Your Screen. Almost None of Them Run on Your Machine.
At Build 2026 the PC quietly became an agent that sees everything you do. The part nobody's pricing in: your screen is the most sensitive surface you own, and the default architecture streams it to someone else's cloud.
At Build 2026 a few weeks ago, Microsoft stopped calling it a PC and started calling it an agentic operating system. The headline launch was Scout, an always-on personal agent that watches your screen across Microsoft 365 and acts on what it sees. It wasn't an outlier. Within a few months in early 2026, every major lab shipped a way for an AI to see your screen, move your mouse, and click. "AI assistant" became "AI agent," and the thing they all agreed on is that the agent should be looking at your display.
Here's the part the launch keynotes skip: a screenshot of your screen is the most sensitive payload in computing. Not your search history, your screen. It's your open DMs, your banking tab, the half-written resignation email, the client's data, the password manager mid-reveal. And the default architecture for all of this new screen-aware AI is to take that frame and send it somewhere else to be understood.
The trust story for a screen-reader is "where does the pixel go"
Once you frame it that way, the only question that matters about a screen-watching assistant is a boring infrastructure one: does the screenshot leave the machine?
For almost every shiny tool shipping right now, the answer is yes. The frame goes to a cloud model. The transcript of what you said goes to a hosted speech API. The whole loop is brokered through someone's proxy, often with your API keys sitting on their server. You're trusting not just that the company is honest, but that their breach surface is clean, forever.
It usually isn't. The cautionary tale here is Cluely, the screen-reading "cheat on everything" assistant that leaked 83k+ users in 2025. When the product's whole job is to watch your screen, a breach isn't an email list, it's a window into everything those users were doing. The trust story for this category can't be a privacy policy. It has to be architecture.
Local-first isn't a slogan, it's a parts list
The reason most tools go to the cloud isn't malice, it's convenience: hosted STT, a frontier vision model, and hosted TTS are three easy API calls. To keep the loop on-device you have to replace each one with something that runs locally and is actually good enough. As of mid-2026, you can:
- Speech-to-text →
whisper.cppon Metal, or Apple's built-inSpeechTranscriberas a zero-dependency fallback. No audio leaves the mic. - Reasoning + vision → the Claude Code CLI as a single brain abstraction. Because it speaks the Anthropic Messages API, the same binary can point at your subscription or at a local Ollama model (
ANTHROPIC_BASE_URL=http://localhost:11434, a vision model likeqwen3-vl:8b). One env var flips between "best quality" and "fully offline." - Text-to-speech → Kokoro running in MLX, fast enough for real-time replies.
The frontier-vs-local gap that made this impractical a year ago has mostly closed for this use case. The brain doesn't need to be GPT-class to read a screen and decide where to click, it needs to be grounded.
The grounding trick, because small models miss
This is the bit I find genuinely fun. A smaller local model is worse at saying "click at pixel 847, 312", vision coordinate grounding is exactly where it's weakest. So you don't trust the pixels. The model picks a target by label, and you resolve that label against the macOS Accessibility tree (AXUIElement) to get the element's real on-screen frame. The model says "the Send button"; the OS tells you precisely where the Send button is. The cursor lands on the element even when the model's raw coordinates would have missed by 40 pixels. Local stays accurate.
So I built one, and it's open source
I've been building exactly this: Loky, a local-first macOS screen companion. It's a fork of Farza's excellent Clicky, Clicky nails the overlay, the pointing, and the push-to-talk; I rewired the cloud loop to run on the machine. Push-to-talk, on-device transcription, a multi-monitor screenshot to a local or subscription brain, and an overlay that flies the cursor to the AX-resolved target and talks back. No hosted proxy, no per-user keys, no frame leaving the Mac unless you deliberately choose a cloud brain.
It's MIT, same as upstream: github.com/ashlocklabs/loky.
The next year of computing is going to ship a lot of software whose entire job is to watch what you do. Some of it will be genuinely useful. The only ones I'd actually leave running are the ones where I can answer the boring question, does the screenshot leave the machine?, with a no. Convenience centralizes. Trust runs local.