ChatGPTClaudeGrok / X / CursorGemini

Testing a DSH plugin without a model: four layers

Plugin DevelopmentPublished 2026-11-05Author: DeepSeek Plugin Market
DSH pluginplugin testingCIdump-configDeepSeek Harness
No model and no UI, yet you can still tell whether a plugin is broken. Four layers, each with a command, a pass condition, and a CI assertion.

The hard part of plugin testing is not writing assertions. It is knowing where they belong. The page opens, the tool clicks, everything looks right — then another machine fails to load it.

Split validation into four layers, each answering one question, and testing stops depending on intuition.

Define a broken DSH plugin first

A plugin can break in four ways, and each maps to a layer. Get the classification straight before choosing commands:

FailureSymptomLayer
Will not loadStartup error or a stuck pluginAssembly
Loads with wrong configBehavior differs from expectationsConfig
Tool errors or renders badlyModel-side error, blank cardContract
Behavior stacks after reloadDoubled timers, duplicate listenersUnload

Only the first blocks startup; the other three mean "runs but wrong". Verify in that order: make it run, then make it correct.

Assembly: does the DSH plugin load

Assembly passes only when the Loader actually mounts and runs the DSH plugin.

One question only: did the Loader mount your plugin?

  1. Build a cordis.yml containing just this plugin and start with --patch;
  2. Log something explicit on the first line of apply, such as [hello] loaded;
  3. Confirm the line appears in the terminal;
  4. Record the exit code; a healthy start should not end with 1;
  5. Delete the plugin file, start again, and confirm the failure names the missing module.

Step 5 is the actual test. Verifying only the success path verifies nothing about failure handling; break it on purpose once so you know what failure looks like.

Do not read only the last terminal line. A required plugin failing to activate writes startup-<timestamp>-<uuid>.log, which contains the plugin states and stack.

Config: are the combined DSH plugin values right

Config bugs hide because defaults make wrong values look normal. Use --dump-config, which does not start the service:

  1. Run <command> --dump-config and find your plugin's layer;
  2. Confirm config holds what you wrote, not a value overridden by a higher patch;
  3. Pass an invalid value, such as a string where a number belongs, and confirm loading fails on that field;
  4. Remove a required field and confirm failure rather than a silent empty value;
  5. Restore the config and confirm startup.

Steps 3 and 4 prove the schema is active. If invalid values still load, either Config was not exported or the schema is a plain object the framework never validated.

One more trap: a patch replaces the whole config row rather than deep-merging. Changing a single key in a profile may wipe other defaults, so test a partial config as well.

Contract: DSH plugin declarations versus behavior

Tools are the most failure-prone part because they span the model, framework, and UI. Four checks need no model:

  1. parameters is inferred correctly and wrong types are rejected before execution;
  2. execute returns a value satisfying output.schema;
  3. render depends only on arguments and the canonical value, reading no external mutable state;
  4. The tool name does not collide with an existing one, and registration rejects duplicates.

Only the third needs your judgment. Render is required to be pure; if it reads a global or the current time, the same result renders differently at different moments and tests cannot reproduce it.

The minimal contract test is "fail once". Return a value violating the schema and confirm an error; register a duplicate name and confirm rejection. Both failing as expected means the contract is actually enforced.

Unload: what a DSH plugin leaves after reload

This layer decides whether the plugin stays maintainable.

  1. Start and record a resource signature, such as listener count or poll frequency;
  2. Trigger at least three hot reloads;
  3. Confirm the signature is unchanged;
  4. Unload and confirm resources disappear, with no port or handle residue;
  5. Reload and confirm the feature returns without "already in use" errors.

Three reloads is deliberate. One reload hides most duplicate-registration bugs; by the third, leaks show up as visibly doubled frequency.

Into CI: four DSH plugin assertions

CI has no UI and no model, so assert four things:

  1. The process starts: the command does not exit within the timeout, or exits 0;
  2. Assembly succeeded: the log contains your load marker;
  3. Config parses: --dump-config exits 0 and its output includes the expected fields;
  4. Failure is detectable: an invalid config exits non-zero.

The fourth is the important one. A CI that asserts only the success path still passes when the plugin never loads, because "did not start" and "started but printed nothing" both look like "the command ended".

A usable skeleton:

bash
export DSH_HOME="$(mktemp -d)"
dsh --profile web --patch ./cordis.yml --dump-config > /tmp/cfg.yml || exit 1
dsh --profile web --patch ./cordis.yml > /tmp/run.log 2>&1 &
grep -q '\[hello\] loaded' /tmp/run.log || exit 1

Point DSH_HOME at a temporary directory. Never use the default ~/.dsh in CI, or one run pollutes the cache and profiles and the next result is incomparable.

One question still needs a real model: will the model select your tool? That depends on the name, description, parameter docs, and system prompt — discoverability, not assembly or contract. Verify it with a natural-language instruction and watch whether it fires; if not, edit the description and naming, not the logic. It is slow, needs a key, and varies, so keep it as a manual spot check rather than a per-commit job.

The goal is not coverage but moving "only the user could find this" earlier. One failing case per layer beats a hundred assertions that always pass. For diagnosing an existing failure, read local debugging; for tool contract details, read the execute contract; for a key-free lab, read the Cordis lab.

ChatGPTClaudeGrok / X / CursorGemini

FAQ

Do DSH plugin tests require a real model, or can I verify anything without one?

Without a model you can verify four things: the plugin loads, its config is valid, the tool definition honors its contract, and unload leaves no residue. Only whether a model selects the tool needs a real model.

What is the fastest local check that a plugin loads, short of opening the UI?

Mount it with --patch and watch the startup log for the plugin's own load line. On failure the process exits non-zero and writes a startup diagnostic; log plus exit code is faster than hunting for an entry in the interface.

How do I verify a DSH plugin tool's parameters and return value match its declaration?

A DSH plugin tool is verified in three places: defineTool infers parameters correctly, execute returns a value satisfying output.schema, and render is a pure function. The framework checks the first two; purity is your responsibility.

What failure signal does plugin validation produce in CI without a UI?

Two kinds: a required plugin failing to activate exits with status 1 and writes a startup diagnostic; assembly succeeding but behavior deviating depends on your own assertion, such as a missing log line. Cover both.

Related Terms

assembly check
An assembly check is a DSH plugin check confirming that the Loader loaded the plugin and it reached a running state, judged by startup logs and process exit code.— DeepSeek Harness docs
contract check
A contract check is a DSH plugin check verifying that a tool's parameter declaration, return value, and render function follow framework conventions.— DeepSeek Harness docs
dump-config
dump-config is a DSH plugin CLI flag that prints the combined configuration tree without starting the service, showing effective values after all patch layers.— DeepSeek Harness CLI reference
diagnostic file
A diagnostic file is the startup log the CLI writes on failure, containing versions, profile, inactive plugins, and raw errors under DSH_HOME's logs directory.— DeepSeek Harness CLI reference

Sources