ChatGPTClaudeGrok / X / CursorGemini

DSH plugin reference tests: no model, no key, still valid

Plugin DevelopmentPublished 2026-11-10Author: DeepSeek Plugin Market
DSH pluginDeepSeek Harnesssession referencetesting64 KiB
CI has no model or key, yet the reference chain needs coverage. The 64 KiB fallback gives a fixed budget: fixtures, four assertion types, and a case table.

"You need a real model to test this" is the most common excuse around the reference chain. In practice, DSH plugin computes a 64 KiB budget when model capacity or an adapter is missing, which gives CI a completely deterministic input.

DeepSeek Harness: Why a no-model test is possible

The 64 KiB fallback turns "depends on the model" into "depends on a fixed number". (source) Causal chain:

  1. The budget prefers the last assembled route;
  2. Without a route, it uses agent options;
  3. Without capacity or an adapter, it uses 64 KiB.

The third is exactly the test environment's state. It exists for incompleteness, but it makes "run references without a key" a legitimate path.

So CI can assert that the same fixture and the same reference order produce identical omission sizes and truncation state every run.

DeepSeek Harness: Building fixed fixtures

Fixtures should be small, stable, and boundary-triggering, not realistic. (source) Three suffice:

FixtureTraitUse
short3 messages, far below budgetNormal path
longMessage volume exceeds 64 KiBTruncation and counts
compactedA source session with a compaction markerSource-side gap

Each fixture carries one boundary; do not try to cover everything with one large session.

Content only needs a few identifiable strings, such as a sequence number per message, so ordering is easy to assert.

Pin metadata such as creation time and cwd, or candidate ordering drifts across machines.

DeepSeek Harness: Four assertion types

Four kinds cover success, failure, and lifecycle. (source) Checklist:

  1. Shape — content is still a block array with mention tokens removed;
  2. Counts — originalMessages = retainedMessages + omittedMessages, with omittedBytes above zero;
  3. Codes — failure cases compare SESSION_REFERENCE_*, never copy;
  4. No publish — cancellation asserts additionalContext is absent.

Type two is cheap and catches bugs sharply: break the recording logic and the three numbers immediately disagree.

Type three must compare the code field, not a serialized error string.

Type four has to be written separately: many suites test "an error was returned" but never "no context was published".

DeepSeek Harness: A case table

Seven cases cover most of the seven codes plus the main flow. (source) Comparison:

CaseInputAssertion
Single referenceshortExactly one additionalContext
Several referencesshort × 3Still at most one aggregated context
Zero referencesempty referencesNo context, content unchanged
Self referencecurrent session idSELF_REFERENCE
Over the limitmore references than allowedTOO_MANY
Over budgetlongtruncated true with consistent counts
Cancellationsignal triggeredCANCELLED with no context

The third row is the easiest to omit: zero references should produce neither an empty context nor an error.

Rows four and five are input validation and trigger with minimal data, no environment faults required.

DeepSeek Harness: Stable cancellation tests

Trigger the signal at a fixed point; do not race with sleep. (source) Approach:

  1. Provide a read implementation that can pause at a chosen stage;
  2. Abort while prepare is running;
  3. Assert SESSION_REFERENCE_CANCELLED;
  4. Assert that no additionalContext is published afterward.

Step 4 is the point: cancellation means "do not publish", not merely "return an error".

If the implementation contains "write the context after storage finishes", this case fails immediately, which is exactly its value.

Never build the race with a fixed setTimeout: it passes intermittently on slow machines and fails intermittently on fast ones.

What not to do

Five practices make tests look green while proving nothing. Checklist:

  • Asserting full prompt text — one wording change turns it red;
  • Mocking prepare itself — you test the mock, not the chain;
  • Depending on real session ids — they vanish in another environment;
  • Judging truncation by wall-clock time — results vary by machine;
  • Testing only the success path — none of the seven codes is covered.

The second is the most dangerous: coverage looks impressive while every regression slips through.

The first and fourth are the same mistake in different clothes: putting something nondeterministic into an assertion.

DeepSeek Harness: CI rollout order

Run three essentials first, then add boundaries. (source) Suggested order:

  1. Single reference plus zero references, covering the main flow and empty input;
  2. Over-budget truncation, covering counts and truncated;
  3. Self reference, over-limit, and cancellation as failure cases.

All three groups run without a key and finish quickly, so they suit every commit.

Once fixtures and assertions are stable, add branches for source compaction and format versions, so too many variables do not arrive at once.

When tests are stable, return to the fields themselves. Read the DSH plugin reference budget and 64 KiB fallback; to design assertions by error code, read the seven DSH plugin reference error codes; for plugin testing overall, read how to test a DSH plugin.

Sources: DeepSeek Harness reference - Session references, session-reference/src/index.ts

ChatGPTClaudeGrok / X / CursorGemini

FAQ

In DSH plugin, do session reference tests need a real model?

No, DSH plugin tests can run without a real model. Without model capacity or an adapter, the automatic budget is 64 KiB, a deterministic value that suits stable assertions in a no-model environment.

For DSH plugin, why not assert on prompt text in reference tests?

Text changes with implementation and wording, so DSH plugin tests should assert stable contracts instead. DSH plugin's stable contract is the field shape, count relationships, and error codes.

With DSH plugin, cancellation tests are flaky in CI. How should they be written?

Trigger the DSH plugin signal at a fixed point rather than relying on timing. DSH plugin publishes no context after cancellation, so assert that additionalContext is absent without waiting on real delays.

In DSH plugin, should test fixtures contain real conversation content?

No, DSH plugin fixtures do not need real conversation content. Fixtures only need the session structure and a few identifiable strings; DSH plugin assertions target shape and counts, not dialogue text.

Related Terms

64 KiB fallback
The 64 KiB fallback is the fixed automatic budget used when model capacity or an adapter is missing, making preparation predictable without a model.— DeepSeek Harness reference - Session references
fixed fixture
A fixed fixture is a prebuilt session with stable content so reference results can be compared repeatably.— session-reference/src/index.ts
shape assertion
A shape assertion checks return structure and types, such as content being a block array and at most one additionalContext.— DeepSeek Harness reference - Session references
no-publish assertion
A no-publish assertion verifies that cancellation placed no additionalContext into the message.— DeepSeek Harness reference - Session references

Sources