mcp-video-analyzer

mcp-video-analyzer

Turn any video — YouTube, Instagram, TikTok, Loom, X, Vimeo, direct links, local files — into transcripts, key frames, OCR text, and metadata for AI agents.

No existing video MCP combines transcripts + visual frames + metadata in one tool. This one does — across Loom, the major yt-dlp platforms (YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook), direct video URLs, and local files.

Want a full pipeline, not just a tool? social-knowledge-base is built on top of this server — it downloads whole Instagram creator accounts (reels, stories, highlights), transcribes them, and turns the result into a searchable, RAG-queryable knowledge base with AI-generated notes. Use this MCP when you want per-video analysis inside an agent; use social-knowledge-base when you want to archive and query an entire account.

Installation

Prerequisites

• Node.js 18+ — required to run the server via npx
• yt-dlp — required for YouTube/Vimeo/TikTok/Instagram/X/Twitch/Dailymotion/Facebook URLs; optional for everything else (improves Loom download quality). Install with pip install yt-dlp
• Chrome/Chromium (optional) — fallback for frame extraction if yt-dlp is unavailable

Without yt-dlp or Chrome, direct URLs and local files still get frames — the bundled ffmpeg-static does the extraction, and Loom falls back to its own CDN download. Platform URLs (YouTube etc.) degrade to a clear "install yt-dlp" warning. Transcripts, metadata, and comments never require either.

There are three ways in: the /video plugin (Claude Code — slash command + MCP server auto-configured), a plain MCP server config (any MCP client), or the portable skill + CLI (Codex, Cursor, Copilot, and any agent with a shell — no MCP required).

Claude Code — /video plugin (recommended)

/plugin marketplace add guimatheus92/mcp-video-analyzer
/plugin install video@mcp-video-analyzer

This adds the /video slash command and auto-registers the MCP server — no claude mcp add needed:

/video https://youtu.be/jNQXAC9IVRw what happens at 0:10?
/video ~/Movies/screen-recording.mp4 when does the UI break?

Other agents — Codex, Cursor, Copilot, Gemini CLI, …

npx skills add guimatheus92/mcp-video-analyzer

Installs the video skill (Agent Skills format) into every agent detected on your machine. Agents without the MCP server configured fall back to the bundled CLI automatically — zero configuration.

Claude Code (MCP only)

claude mcp add video-analyzer -- npx mcp-video-analyzer@latest

Then restart Claude Code or start a new conversation.

VS Code / Cursor

Add to your MCP settings file:

• VS Code: File → Preferences → Settings → search "MCP" or edit ~/.vscode/mcp.json / %APPDATA%\Code\User\mcp.json (Windows)
• Cursor: Settings → MCP Servers → Add

{
  "servers": {
    "mcp-video-analyzer": {
      "type": "stdio",
      "command": "npx",
      "args": ["mcp-video-analyzer@latest"]
    }
  }
}

Then reload the window (Ctrl+Shift+P → "Developer: Reload Window").

Claude Desktop

Add to your Claude Desktop config file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "video-analyzer": {
      "command": "npx",
      "args": ["mcp-video-analyzer@latest"]
    }
  }
}

Then restart Claude Desktop.

CLI (one-shot, no MCP client)

The same engine is exposed as a one-shot command — this is what the video skill uses on agents without MCP, and it works standalone in any terminal:

npx -y mcp-video-analyzer@latest analyze "https://youtu.be/jNQXAC9IVRw"

stdout is a single JSON document — metadata, transcript, ocrResults, timeline, warnings, frameCount, and frames as { time, filePath, mimeType } entries pointing at JPEG key frames copied to --out (default: <tmp>/mcp-video-analyzer/<url-hash>/). Progress streams on stderr, so stdout can be piped straight into a JSON parser. Partial failures land in warnings with exit code 0; only hard failures exit 1.

Run with no arguments (npx mcp-video-analyzer@latest) to start the MCP stdio server — the CLI is purely additive.

Verify it works

Once installed, ask your AI assistant:

Analyze this video: https://www.youtube.com/watch?v=jNQXAC9IVRw

(also works with an Instagram/TikTok/Loom link, a direct .mp4 URL, or a local file path). If the server is connected, it will automatically call the analyze_video tool.

Tools

Eight tools — the AI picks the cheapest one for the job and calls it automatically. Click any tool to expand its parameters and examples.

> Analyze this video: https://www.youtube.com/watch?v=abc123...

> Analyze every .mp4 in this folder

> Get the transcript from this video

> What's this video about?

> Extract frames from this video with dense sampling

> Analyze what happens between 1:30 and 2:00 in this video

> Show me the frame at 1:23 in this video

> Show me 10 frames between 0:15 and 0:17 of this video

Detail Levels

Caching

Results are cached in memory for 10 minutes. Subsequent calls with the same URL and options return instantly. Use forceRefresh: true to bypass the cache.

Persistent sidecars (resumable bulk processing)

The in-memory cache is lost on restart, which makes reprocessing a large local corpus costly. Set MCP_WRITE_SIDECARS=1 to also persist results next to each local video so the work survives restarts and can resume:

• <stem>.vtt — the transcript, only when it was generated by the Whisper fallback (an existing <stem>.vtt from your own pipeline is never overwritten). A later call reuses it via the normal sidecar reader and skips Whisper entirely.
• <stem>.analysis.json + <stem>.frames/ — the full result (frames + OCR + timeline), keyed by the video's mtime:size and the analysis params. On a later call with a matching stamp + params, the result is returned straight from disk (no extraction, no OCR).

This makes analyze_videos over thousands of files resumable, and lets an external GPU transcription pipeline and this MCP share results through the filesystem: the pipeline writes <stem>.vtt, and the MCP picks it up instead of running Whisper.

Supported Sources

Local files: pass an absolute path (e.g., /Users/you/clip.mp4) or a file:// URI as the url argument to any tool. Relative paths are rejected — the server's working directory is unpredictable from the MCP client. Note that any caller of the MCP server can ask it to read any file the server process has access to.

Sidecar transcripts: if a clip.vtt, clip.srt, clip.en.vtt, etc. lives next to clip.mp4, it's used as the transcript automatically — no Whisper roundtrip needed. SRT is converted to VTT in-memory.

Embedded subtitles: if no sidecar is found and the container has an embedded subtitle stream (common in .mkv / .mov / .mp4 from screen recorders), it's transmuxed to VTT via ffmpeg and used as the transcript.

Recognized extensions (local files and direct URLs): .mp4 .mov .mkv .webm .avi .m4v .wmv .flv .mpeg .mpg .m2ts .mts .3gp .ogv. The extension only gates routing — ffmpeg does the actual demuxing, so most common containers work. .ts is excluded to avoid colliding with TypeScript source files.

Platform URLs via yt-dlp (YouTube, Instagram, TikTok, …)

Single-video pages on major platforms route through yt-dlp (pip install yt-dlp — required for these URLs). Playlists, channels, and profile pages are rejected by design; pass individual video URLs (batch them with analyze_videos).

• Transcript: native captions are preferred and free — uploaded subtitles first, auto-generated captions as fallback (rolling-window duplication is collapsed). WHISPER_LANGUAGE (e.g. pt) is also used to pick the caption language. Videos with no captions at all fall through to the normal Whisper chain.
• Metadata: title, duration, uploader/channel, view count, upload date, and chapters — no download needed.
• Download: capped at 1080p (frames/OCR don't need more), live streams are skipped, and DASH audio+video is merged with the bundled ffmpeg-static (no system ffmpeg required).
• Cookies — Instagram and age-restricted videos usually require a logged-in session:

Browser cookie extraction requires the browser to be closed on Windows (the cookie database is locked while it runs). If that's inconvenient, export a cookies.txt once (e.g. with a "Get cookies.txt" browser extension) and point YTDLP_COOKIES at it. Private/age-restricted videos without valid cookies don't crash the tool — the yt-dlp ERROR: line surfaces in warnings[].

TwelveLabs Pegasus (optional)

Set the TWELVELABS_API_KEY environment variable to analyze direct video URLs with TwelveLabs Pegasus. Pegasus analyzes the video server-side (visuals and its own audio) and returns an AI-generated, timestamped transcript plus an AI summary as text — capabilities the DirectAdapter can't provide (a raw .mp4 URL has no transcript or summary on its own), and with no Whisper key required.

The transcript is best-effort LLM output, not a deterministic ASR dump: Pegasus is prompted to emit [MM:SS] line rows, and lines that don't match that shape are dropped, so wording and exact timestamps depend on the model's prompt adherence. Failures (bad key, timeout, API error) surface in the tool's warnings[] rather than silently returning an empty transcript.

The biggest win is on the text-only paths: get_transcript and get_metadata return a Pegasus transcript and summary for direct URLs — a few KB of text, no frame images, no per-frame token cost. analyze_video at detail: "standard"/"detailed" still extracts frames in addition (use detail: "brief" to stay text-only).

Long videos: the summary and full transcript share a single capped completion (max_tokens = 16384), so for very long videos the transcript may be truncated. For multi-hour content, chunking by time window is the better approach.

It's fully opt-in and non-breaking: when TWELVELABS_API_KEY is set the TwelveLabsAdapter handles direct video URLs (it registers the public URL with TwelveLabs — no upload); when it's unset, the DirectAdapter handles them exactly as before. Loom URLs are unaffected. Get a key at playground.twelvelabs.io.

Transcription (Whisper fallback)

When a source has no native transcript (no sidecar .vtt/.srt, no embedded subtitles, no platform captions), the audio track is transcribed with Whisper via a graceful fallback chain (in execution order):

Silent tracks: before any Whisper run, the audio is probed with ffmpeg volumedetect (first 2 minutes). A present-but-mute track — common in muted Reels/Stories — skips transcription entirely and emits a warning that the empty transcript is expected content, not an error, saving a pointless Whisper run.

1. @huggingface/transformers (JS-native, zero external deps) — opt-in only: this strategy runs first, but only when WHISPER_HF_MODEL is explicitly set. When it's unset (the default) the strategy is skipped entirely, so the CLI below wins and its WHISPER_MODEL/WHISPER_LANGUAGE settings are never silently overridden.
2. whisper CLI — used when a whisper executable is found (pip install -U openai-whisper). Point WHISPER_BIN at the executable if it isn't on PATH. Model via WHISPER_MODEL, language via WHISPER_LANGUAGE. The bundled ffmpeg-static is put on the CLI's PATH automatically, so no system ffmpeg is required.
3. OpenAI Whisper API — used when OPENAI_API_KEY is set.

No backend configured? If none of the three is available (no whisper on PATH/WHISPER_BIN, no OPENAI_API_KEY, no WHISPER_HF_MODEL), transcription tools return an empty transcript with a warning telling you how to enable one — rather than a silent "no transcript". Install openai-whisper or set one of the keys above. (The CLI is spawned with PYTHONUTF8=1 so non-English/CJK transcripts don't crash the Python process on Windows.)

The default tiny model is fast but weak for non-English audio. For Portuguese (or other non-English) sources, install the CLI and set WHISPER_MODEL=small (or medium) + WHISPER_LANGUAGE=pt for much better accuracy. Add WHISPER_PROMPT with a domain glossary (brand/place names) to fix proper nouns. You can also override model/language/initialPrompt per call on analyze_video / get_transcript / analyze_videos — no restart needed.

GPU (faster-whisper): whisper-ctranslate2 (pip install -U whisper-ctranslate2) is a drop-in CLI with the same flags plus --device cuda / --compute_type / --beam_size. Point WHISPER_BIN at it and set WHISPER_DEVICE=cuda (+ optionally WHISPER_COMPUTE=float16). These GPU flags are env-gated — they're only passed when set, so plain openai-whisper (which rejects --compute_type) keeps working when they're unset.

Windows note: pip installs whisper.exe into the Python Scripts/ dir, which is often not on the PATH that GUI-launched MCP clients inherit. If transcripts come back empty, set WHISPER_BIN to the full path of whisper.exe.

Frame Extraction Strategies

Frame extraction uses a two-strategy fallback chain — no single dependency is required:

The fallback is automatic — if yt-dlp is not available, the server tries browser-based extraction via puppeteer-core. If neither is available, analysis still returns transcript + metadata + comments, just no frames.

Post-Processing Pipeline

After frame extraction, the pipeline automatically applies:

The OCR step requires tesseract.js (included as a dependency). If it fails to load, analysis continues without OCR — no frames or transcript are lost. OCR preprocessing is on by default; set MCP_OCR_PREPROCESS=0 to OCR the raw frames instead.

Complementary Tools

Chrome DevTools MCP

For live web debugging alongside video analysis, pair this server with the Chrome DevTools MCP:

claude mcp add chrome-devtools npx @anthropic-ai/mcp-devtools@latest

When to use each:

The two MCPs complement each other: video analyzer understands recorded content, DevTools interacts with live pages.

Example Output

The examples/loom-demo/ folder contains real outputs from analyzing a public Loom video (Boost In-App Demo Video, 2:55).

Frame images (19 total in examples/loom-demo/frames/):

• scene_*.jpg — scene-change detection (key visual transitions)
• dense_*.jpg — 1fps dense sampling (every 10th frame saved as sample)
• burst_*.jpg — burst extraction for moment analysis (0:30–0:45)

Regenerate after changes: npx tsx examples/generate.ts — requires yt-dlp + network access.

Development

# Install dependencies
npm install

# Run all checks (format, lint, typecheck, knip, tests)
npm run check

# Build
npm run build

# Run E2E tests (requires network)
npm run test:e2e

# Open MCP Inspector for manual testing
npm run inspect

Architecture

src/
├── index.ts                    # Entry point (shebang + stdio)
├── server.ts                   # FastMCP server + tool registration
├── tools/                      # MCP tool definitions (7 tools)
│   ├── analyze-video.ts        # Full analysis with detail levels + caching
│   ├── analyze-moment.ts       # Deep-dive on a time range
│   ├── get-transcript.ts       # Transcript-only with Whisper fallback
│   ├── get-metadata.ts         # Metadata + comments + chapters
│   ├── get-frames.ts           # Frames-only (scene-change or dense)
│   ├── get-frame-at.ts         # Single frame at timestamp
│   └── get-frame-burst.ts      # N frames in a time range
├── adapters/                   # Source-specific logic
│   ├── adapter.interface.ts    # IVideoAdapter interface + registry
│   ├── loom.adapter.ts         # Loom: authless GraphQL
│   ├── local-file.adapter.ts   # Local files: absolute path or file:// URI
│   ├── twelvelabs.adapter.ts   # TwelveLabs Pegasus: transcript + AI summary (opt-in)
│   └── direct.adapter.ts       # Direct URL: any mp4/webm link
├── processors/                 # Shared processing
│   ├── frame-extractor.ts      # ffmpeg scene detection + dense + burst extraction
│   ├── browser-frame-extractor.ts # Headless Chrome fallback for frames
│   ├── audio-transcriber.ts    # Whisper fallback (HF transformers → CLI → OpenAI)
│   ├── image-optimizer.ts      # sharp resize/compress
│   ├── frame-dedup.ts          # Perceptual dedup (dHash + Hamming distance)
│   ├── frame-ocr.ts            # OCR text extraction (tesseract.js)
│   └── annotated-timeline.ts   # Unified timeline (transcript + frames + OCR)
├── config/
│   └── detail-levels.ts        # brief / standard / detailed config
├── utils/
│   ├── cache.ts                # In-memory TTL cache with LRU eviction
│   ├── field-filter.ts         # Selective field filtering for responses
│   ├── url-detector.ts         # Platform detection from URL
│   ├── vtt-parser.ts           # WebVTT → transcript entries
│   └── temp-files.ts           # Temp directory management
└── types.ts                    # Shared TypeScript interfaces

License

MIT