Testing a DSH plugin without a model: four layers
The hard part of plugin testing is not writing assertions. It is knowing where they belong. The page opens, the tool clicks, everything looks right — then another machine fails to load it.
Split validation into four layers, each answering one question, and testing stops depending on intuition.
Define a broken DSH plugin first
A plugin can break in four ways, and each maps to a layer. Get the classification straight before choosing commands:
| Failure | Symptom | Layer |
|---|---|---|
| Will not load | Startup error or a stuck plugin | Assembly |
| Loads with wrong config | Behavior differs from expectations | Config |
| Tool errors or renders badly | Model-side error, blank card | Contract |
| Behavior stacks after reload | Doubled timers, duplicate listeners | Unload |
Only the first blocks startup; the other three mean "runs but wrong". Verify in that order: make it run, then make it correct.
Assembly: does the DSH plugin load
Assembly passes only when the Loader actually mounts and runs the DSH plugin.
One question only: did the Loader mount your plugin?
- Build a
cordis.ymlcontaining just this plugin and start with--patch; - Log something explicit on the first line of
apply, such as[hello] loaded; - Confirm the line appears in the terminal;
- Record the exit code; a healthy start should not end with 1;
- Delete the plugin file, start again, and confirm the failure names the missing module.
Step 5 is the actual test. Verifying only the success path verifies nothing about failure handling; break it on purpose once so you know what failure looks like.
Do not read only the last terminal line. A required plugin failing to activate writes startup-<timestamp>-<uuid>.log, which contains the plugin states and stack.
Config: are the combined DSH plugin values right
Config bugs hide because defaults make wrong values look normal. Use --dump-config, which does not start the service:
- Run
<command> --dump-configand find your plugin's layer; - Confirm
configholds what you wrote, not a value overridden by a higher patch; - Pass an invalid value, such as a string where a number belongs, and confirm loading fails on that field;
- Remove a required field and confirm failure rather than a silent empty value;
- Restore the config and confirm startup.
Steps 3 and 4 prove the schema is active. If invalid values still load, either Config was not exported or the schema is a plain object the framework never validated.
One more trap: a patch replaces the whole config row rather than deep-merging. Changing a single key in a profile may wipe other defaults, so test a partial config as well.
Contract: DSH plugin declarations versus behavior
Tools are the most failure-prone part because they span the model, framework, and UI. Four checks need no model:
parametersis inferred correctly and wrong types are rejected before execution;executereturns a value satisfyingoutput.schema;renderdepends only on arguments and the canonical value, reading no external mutable state;- The tool name does not collide with an existing one, and registration rejects duplicates.
Only the third needs your judgment. Render is required to be pure; if it reads a global or the current time, the same result renders differently at different moments and tests cannot reproduce it.
The minimal contract test is "fail once". Return a value violating the schema and confirm an error; register a duplicate name and confirm rejection. Both failing as expected means the contract is actually enforced.
Unload: what a DSH plugin leaves after reload
This layer decides whether the plugin stays maintainable.
- Start and record a resource signature, such as listener count or poll frequency;
- Trigger at least three hot reloads;
- Confirm the signature is unchanged;
- Unload and confirm resources disappear, with no port or handle residue;
- Reload and confirm the feature returns without "already in use" errors.
Three reloads is deliberate. One reload hides most duplicate-registration bugs; by the third, leaks show up as visibly doubled frequency.
Into CI: four DSH plugin assertions
CI has no UI and no model, so assert four things:
- The process starts: the command does not exit within the timeout, or exits 0;
- Assembly succeeded: the log contains your load marker;
- Config parses:
--dump-configexits 0 and its output includes the expected fields; - Failure is detectable: an invalid config exits non-zero.
The fourth is the important one. A CI that asserts only the success path still passes when the plugin never loads, because "did not start" and "started but printed nothing" both look like "the command ended".
A usable skeleton:
export DSH_HOME="$(mktemp -d)"
dsh --profile web --patch ./cordis.yml --dump-config > /tmp/cfg.yml || exit 1
dsh --profile web --patch ./cordis.yml > /tmp/run.log 2>&1 &
grep -q '\[hello\] loaded' /tmp/run.log || exit 1
Point DSH_HOME at a temporary directory. Never use the default ~/.dsh in CI, or one run pollutes the cache and profiles and the next result is incomparable.
One question still needs a real model: will the model select your tool? That depends on the name, description, parameter docs, and system prompt — discoverability, not assembly or contract. Verify it with a natural-language instruction and watch whether it fires; if not, edit the description and naming, not the logic. It is slow, needs a key, and varies, so keep it as a manual spot check rather than a per-commit job.
The goal is not coverage but moving "only the user could find this" earlier. One failing case per layer beats a hundred assertions that always pass. For diagnosing an existing failure, read local debugging; for tool contract details, read the execute contract; for a key-free lab, read the Cordis lab.
FAQ
Without a model you can verify four things: the plugin loads, its config is valid, the tool definition honors its contract, and unload leaves no residue. Only whether a model selects the tool needs a real model.
Mount it with --patch and watch the startup log for the plugin's own load line. On failure the process exits non-zero and writes a startup diagnostic; log plus exit code is faster than hunting for an entry in the interface.
A DSH plugin tool is verified in three places: defineTool infers parameters correctly, execute returns a value satisfying output.schema, and render is a pure function. The framework checks the first two; purity is your responsibility.
Two kinds: a required plugin failing to activate exits with status 1 and writes a startup diagnostic; assembly succeeding but behavior deviating depends on your own assertion, such as a missing log line. Cover both.
Related Terms
- assembly check
- An assembly check is a DSH plugin check confirming that the Loader loaded the plugin and it reached a running state, judged by startup logs and process exit code.— DeepSeek Harness docs
- contract check
- A contract check is a DSH plugin check verifying that a tool's parameter declaration, return value, and render function follow framework conventions.— DeepSeek Harness docs
- dump-config
- dump-config is a DSH plugin CLI flag that prints the combined configuration tree without starting the service, showing effective values after all patch layers.— DeepSeek Harness CLI reference
- diagnostic file
- A diagnostic file is the startup log the CLI writes on failure, containing versions, profile, inactive plugins, and raw errors under DSH_HOME's logs directory.— DeepSeek Harness CLI reference
Sources
- DeepSeek Harness docs: your first plugin· deepseek-harness
- DeepSeek Harness docs: building a tool· deepseek-harness
- DeepSeek Harness CLI reference· deepseek-ai