dsh-video-storyboard: Let DeepSeek Harness Read a Video as One Image

What Does It Do?

It is a DSH plugin for DeepSeek Harness that packs a whole video into a single numbered storyboard image, so text-only agents can actually look at footage instead of reading a summary. It solves the token cost and blindness of frame-by-frame inspection: a 5-minute clip becomes one image with 9–16 cells for roughly 384 tokens, where a frame-per-image approach spends 3,072–6,144 for the same coverage. Core capabilities include scene-change detection, density-aware cell placement, per-cell timestamps with transcript lines, an audio envelope strip, and a zoom call for close-ups.

Core Features

• Single-image storyboard: lays a video out as 9–16 numbered cells, each with a timestamp range and the transcript line inside it, so the model can cite a specific cell.
• Scene-aware cutting: detects scene changes in HSV space and picks the threshold by target cut rate, so fast-cut footage does not collapse to zero cuts.
• Density-adaptive layout: allocates cells by local cut density, giving dense stretches more cells while a static take is not split into identical frames.
• Sharpest-frame selection: samples three candidates per segment and keeps the highest Laplacian variance, avoiding blurred transition frames.
• Audio envelope and transcript: a footer strip marks loudness, silence and cut points on the same time axis, and the transcript is computed before cell decisions to set each cell's length.
• Local zoom: video_zoom returns a close-up of a chosen cell without reprocessing the whole file.

How to Use This Plugin?

After enabling it in DSH, hand a video file to the video_storyboard tool; it returns one storyboard image plus the timestamp and transcript data for each cell, which the model can cite or follow up on. When detail matters, call video_zoom with a cell number to get an enlarged view. Repeat calls on the same file hit the cache: a 90-second clip scans in about 9 seconds and returns in about 3.6 seconds on the second call. It requires Node 20 or newer plus ffmpeg and ffprobe on the PATH, with no other runtime dependencies; it is maintained by SAOS-3624 under the MIT license.

How to Troubleshoot This Plugin?

When storyboard generation fails, first confirm that ffmpeg and ffprobe can be invoked directly from the command line, which is the most common environment issue. If the cut count looks too low or cells repeat the same content, the threshold does not match the footage's pacing, so adjust the cut-rate settings as described in the official documentation. If returned frames look blurry, check whether a histogram-based frame picker was used instead; this plugin selects the sharpest frame by Laplacian variance by default.