Gridhand

Rust CLI that lets an agent click desktop GUIs by naming a labeled grid cell instead of guessing pixel coordinates.

Works with setup

Test report

Verdict
Works with setup
Score
8.4/10
Tested
Jul 16, 2026
Environment
Claude Code 2.x (agent harness)
Upstream re-checked
Aug 10, 2026 · 8b48409

Real ~9,100-line zero-dependency Rust binary (verified: empty Cargo.toml deps, Cargo.lock has exactly 1 entry, no AT-SPI/AXUIElement/UIAutomation references in source) implementing genuine OS-level screenshot/input via CoreGraphics/uinput/user32 — not a stub. But it needs a Rust toolchain (or a trusted prebuilt release binary) plus OS permission grants (macOS Accessibility+Screen Recording, Linux udev/input-group via sudo) before a single command runs, and this sandbox had neither cargo nor grantable OS permissions, so the actual click-accuracy of the grid method could not be verified end to end.

Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works

  • Installs cleanly 5/5
  • Triggers reliably 5/5
  • Output vs. baseline 6/10
  • Docs & honesty 5/5

What Gridhand does

A zero-dependency, cross-platform (macOS/Linux/Windows) desktop-automation CLI: screenshot, list/raise windows, click, type, and press key combos, all JSON in/out. It never reads the OS accessibility tree — it overlays a labeled grid with crosshairs on the raw screenshot and you zoom/click by cell name, so it also drives canvas/WebGL/game UIs that accessibility-tree tools can't see. Triggers when an agent needs to see the screen or click/type into any desktop app.

How to install Gridhand

git clone https://github.com/ZachRouan/agent-desktop-interface.git
cd agent-desktop-interface && ./setup.sh   # builds the gridhand binary + platform permissions
mkdir -p ~/.claude/skills/gridhand && cp skills/gridhand/SKILL.md ~/.claude/skills/gridhand/SKILL.md

Skills live in ~/.claude/skills/ (global) or .claude/skills/ (per-project). Restart Claude Code after installing.

Commands — how to trigger Gridhand

  • /gridhand Rust CLI that lets an agent click desktop GUIs by naming a labeled grid cell instead of guessing pixel coordinates.

It also activates on plain-language prompts like these:

  • Take a screenshot of my desktop and click the labeled grid cell
  • Automate clicking inside this game's canvas UI for me
  • Type this text into that desktop app window for me

Frequently asked questions

Is the Gridhand skill free?
Yes. The skill itself is free from ZachRouan/agent-desktop-interface. SkillProof publishes the install command and an independent test verdict at no cost.
Does Gridhand work with Claude Code?
We tested it with Claude Code 2.x (agent harness) on Jul 16, 2026. Verdict: Works with setup. Real ~9,100-line zero-dependency Rust binary (verified: empty Cargo.toml deps, Cargo.lock has exactly 1 entry, no AT-SPI/AXUIElement/UIAutomation references in source) implementing genuine OS-level screenshot/input via CoreGraphics/uinput/user32 — not a stub. But it needs a Rust toolchain (or a trusted prebuilt release binary) plus OS permission grants (macOS Accessibility+Screen Recording, Linux udev/input-group via sudo) before a single command runs, and this sandbox had neither cargo nor grantable OS permissions, so the actual click-accuracy of the grid method could not be verified end to end.
What is the Gridhand SkillProof Score?
8.4/10 — installs cleanly 5/5, triggers reliably 5/5, output vs. baseline 6/10, docs & honesty 5/5.
How do I install Gridhand?
Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
Can I use Gridhand with Cursor, Copilot, Gemini CLI, Codex or other AI tools?
The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.