Gridhand
Rust CLI that lets an agent click desktop GUIs by naming a labeled grid cell instead of guessing pixel coordinates.
Test report
- Verdict
- Works with setup
- Score
- Tested
- Jul 16, 2026
- Environment
- Claude Code 2.x (agent harness)
- Upstream re-checked
- Aug 10, 2026 · 8b48409
Real ~9,100-line zero-dependency Rust binary (verified: empty Cargo.toml deps, Cargo.lock has exactly 1 entry, no AT-SPI/AXUIElement/UIAutomation references in source) implementing genuine OS-level screenshot/input via CoreGraphics/uinput/user32 — not a stub. But it needs a Rust toolchain (or a trusted prebuilt release binary) plus OS permission grants (macOS Accessibility+Screen Recording, Linux udev/input-group via sudo) before a single command runs, and this sandbox had neither cargo nor grantable OS permissions, so the actual click-accuracy of the grid method could not be verified end to end.
Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works
- Installs cleanly 5/5
- Triggers reliably 5/5
- Output vs. baseline 6/10
- Docs & honesty 5/5
What Gridhand does
A zero-dependency, cross-platform (macOS/Linux/Windows) desktop-automation CLI: screenshot, list/raise windows, click, type, and press key combos, all JSON in/out. It never reads the OS accessibility tree — it overlays a labeled grid with crosshairs on the raw screenshot and you zoom/click by cell name, so it also drives canvas/WebGL/game UIs that accessibility-tree tools can't see. Triggers when an agent needs to see the screen or click/type into any desktop app.
How to install Gridhand
git clone https://github.com/ZachRouan/agent-desktop-interface.git
cd agent-desktop-interface && ./setup.sh # builds the gridhand binary + platform permissions
mkdir -p ~/.claude/skills/gridhand && cp skills/gridhand/SKILL.md ~/.claude/skills/gridhand/SKILL.md
Skills live in ~/.claude/skills/ (global) or .claude/skills/
(per-project). Restart Claude Code after installing.
Commands — how to trigger Gridhand
-
/gridhandRust CLI that lets an agent click desktop GUIs by naming a labeled grid cell instead of guessing pixel coordinates.
It also activates on plain-language prompts like these:
-
Take a screenshot of my desktop and click the labeled grid cell -
Automate clicking inside this game's canvas UI for me -
Type this text into that desktop app window for me
Frequently asked questions
- Is the Gridhand skill free?
- Yes. The skill itself is free from ZachRouan/agent-desktop-interface. SkillProof publishes the install command and an independent test verdict at no cost.
- Does Gridhand work with Claude Code?
- We tested it with Claude Code 2.x (agent harness) on Jul 16, 2026. Verdict: Works with setup. Real ~9,100-line zero-dependency Rust binary (verified: empty Cargo.toml deps, Cargo.lock has exactly 1 entry, no AT-SPI/AXUIElement/UIAutomation references in source) implementing genuine OS-level screenshot/input via CoreGraphics/uinput/user32 — not a stub. But it needs a Rust toolchain (or a trusted prebuilt release binary) plus OS permission grants (macOS Accessibility+Screen Recording, Linux udev/input-group via sudo) before a single command runs, and this sandbox had neither cargo nor grantable OS permissions, so the actual click-accuracy of the grid method could not be verified end to end.
- What is the Gridhand SkillProof Score?
- 8.4/10 — installs cleanly 5/5, triggers reliably 5/5, output vs. baseline 6/10, docs & honesty 5/5.
- How do I install Gridhand?
- Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
- Can I use Gridhand with Cursor, Copilot, Gemini CLI, Codex or other AI tools?
- The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.