Video Perception
Drives an ffmpeg MCP server so Claude reads video frames and timestamped transcripts
Test report
- Verdict
- Works with setup
- Score
- Tested
- Jul 21, 2026
- Environment
- Claude Code 2.x (agent harness)
- Upstream re-checked
- Aug 10, 2026 · 6eded2f
I installed claude-video-vision@1.3.2 from npm into a sandboxed HOME, drove the MCP server over stdio JSON-RPC, and confirmed all six tools respond, then generated a 3-minute test video with four labelled chapters and a 1-second "ERROR CODE 7742" flash at 01:37. BASELINE (naive video_watch, fps auto): 100 frames / 487KB of images; it caught the flash at 00:01:37 but silently hit max_frames=100 and never extracted the final 41 seconds of the video. SKILL run (video_info then video_analyze then a planned segment extraction): video_analyze returned the exact cut list 00:00:45, 00:01:30, 00:01:37, 00:01:38, 00:02:15 plus the 00:00:45-00:01:30 silence interval for zero image tokens — including the 02:15 cut baseline never saw — but the prescribed segment extraction returned only 34KB of frames labelled 00:00:10, 00:01:48, 00:01:41, 00:02:21, 00:02:34 and 00:03:54 (past the 3:00 duration) and missed the error flash entirely. Root cause I read in dist/extractors/frames.js: -ss is passed before -i while -to is passed after, so every window request runs to EOF; I reproduced it on video_watch segments, video_watch start_time/end_time, and video_detail in a clean HOME. Transcription also returns an empty array with no warning when no whisper/API backend is configured.
Scored on four weighted criteria — install, triggering, output vs. baseline, docs. How scoring works
- Installs cleanly 4/5
- Triggers reliably 5/5
- Output vs. baseline 5/10
- Docs & honesty 3/5
What Video Perception does
Wires Claude Code into the claude-video-vision MCP server so it can read local video files (.mp4, .mov, .mkv, .webm) and YouTube URLs as image frames plus timestamped audio transcription. The skill prescribes a workflow: fetch metadata, run ffmpeg structural analysis (scene changes, silence intervals, motion) before extracting any frames, then pull frames at adaptive FPS around the moments that matter. Triggers when a video file path, a YouTube link, or a request to watch, review, or summarize a video appears in the conversation.
How to install Video Perception
git clone --depth 1 https://github.com/jordanrendric/claude-video-vision.git /tmp/video-perception-src
mkdir -p ~/.claude/skills
cp -R /tmp/video-perception-src/skills/video-perception ~/.claude/skills/video-perception
# The skill is only guidance — it is USELESS without the MCP server that provides
# video_info / video_analyze / video_watch / video_detail / video_configure / video_setup.
#
# Register the MCP server (needs Node 20+ and ffmpeg on PATH):
# claude mcp add claude-video-vision -- npx -y claude-video-vision@latest
#
# Or install the whole plugin instead of this skill, inside Claude Code, one at a time:
# /plugin marketplace add https://github.com/jordanrendric/claude-video-vision
# /plugin install claude-video-vision
# /setup-video-vision
#
# Required deps:
# brew install ffmpeg # frame extraction (mandatory)
# brew install yt-dlp # only for YouTube URLs
# Audio transcription needs a backend, otherwise transcription silently returns []:
# export GEMINI_API_KEY=... # free tier, backend "gemini-api"
# brew install whisper-cpp # fully local, backend "local"
# export OPENAI_API_KEY=... # paid, backend "openai"
# Config lives at ~/.claude-video-vision/config.json (set via the video_configure tool).
#
# Known defect in the published v1.3.2 server: any time-bounded request over-extracts.
# ffmpeg is called with -ss before -i but -to after -i, so asking for 01:36-01:39
# returns ~100 frames spanning 01:36-02:25. Prefer whole-video video_watch and lean on
# video_analyze for timestamps until this is fixed.
Skills live in ~/.claude/skills/ (global) or .claude/skills/
(per-project). Restart Claude Code after installing.
Commands — how to trigger Video Perception
-
/video-perceptionDrives an ffmpeg MCP server so Claude reads video frames and timestamped transcripts
It also activates on plain-language prompts like these:
-
Watch this video and summarize what happens -
Analyze this YouTube link for key moments -
Review this MP4 and describe the scenes
Frequently asked questions
- Is the Video Perception skill free?
- Yes. The skill itself is free from jordanrendric/claude-video-vision. SkillProof publishes the install command and an independent test verdict at no cost.
- Does Video Perception work with Claude Code?
- We tested it with Claude Code 2.x (agent harness) on Jul 21, 2026. Verdict: Works with setup. I installed claude-video-vision@1.3.2 from npm into a sandboxed HOME, drove the MCP server over stdio JSON-RPC, and confirmed all six tools respond, then generated a 3-minute test video with four labelled chapters and a 1-second "ERROR CODE 7742" flash at 01:37. BASELINE (naive video_watch, fps auto): 100 frames / 487KB of images; it caught the flash at 00:01:37 but silently hit max_frames=100 and never extracted the final 41 seconds of the video. SKILL run (video_info then video_analyze then a planned segment extraction): video_analyze returned the exact cut list 00:00:45, 00:01:30, 00:01:37, 00:01:38, 00:02:15 plus the 00:00:45-00:01:30 silence interval for zero image tokens — including the 02:15 cut baseline never saw — but the prescribed segment extraction returned only 34KB of frames labelled 00:00:10, 00:01:48, 00:01:41, 00:02:21, 00:02:34 and 00:03:54 (past the 3:00 duration) and missed the error flash entirely. Root cause I read in dist/extractors/frames.js: -ss is passed before -i while -to is passed after, so every window request runs to EOF; I reproduced it on video_watch segments, video_watch start_time/end_time, and video_detail in a clean HOME. Transcription also returns an empty array with no warning when no whisper/API backend is configured.
- What is the Video Perception SkillProof Score?
- 6.8/10 — installs cleanly 4/5, triggers reliably 5/5, output vs. baseline 5/10, docs & honesty 3/5.
- How do I install Video Perception?
- Copy the install command from this page, run it in your terminal, and restart Claude Code. Skills live in ~/.claude/skills/ (global) or .claude/skills/ inside a project.
- Can I use Video Perception with Cursor, Copilot, Gemini CLI, Codex or other AI tools?
- The SKILL.md format is native to Claude (Claude Code, Desktop, claude.ai). The instructions inside adapt to other assistants: Cursor rules, GitHub Copilot instructions, Windsurf rules, Custom GPTs, AGENTS.md for OpenAI Codex, and GEMINI.md for Google Gemini CLI — our conversion guides cover each, and the free converter on the tools page does the wrapping for you.