DSH plugin reference tests: no model, no key, still valid
"You need a real model to test this" is the most common excuse around the reference chain. In practice, DSH plugin computes a 64 KiB budget when model capacity or an adapter is missing, which gives CI a completely deterministic input.
DeepSeek Harness: Why a no-model test is possible
The 64 KiB fallback turns "depends on the model" into "depends on a fixed number". (source) Causal chain:
- The budget prefers the last assembled route;
- Without a route, it uses agent options;
- Without capacity or an adapter, it uses 64 KiB.
The third is exactly the test environment's state. It exists for incompleteness, but it makes "run references without a key" a legitimate path.
So CI can assert that the same fixture and the same reference order produce identical omission sizes and truncation state every run.
DeepSeek Harness: Building fixed fixtures
Fixtures should be small, stable, and boundary-triggering, not realistic. (source) Three suffice:
| Fixture | Trait | Use |
|---|---|---|
short | 3 messages, far below budget | Normal path |
long | Message volume exceeds 64 KiB | Truncation and counts |
compacted | A source session with a compaction marker | Source-side gap |
Each fixture carries one boundary; do not try to cover everything with one large session.
Content only needs a few identifiable strings, such as a sequence number per message, so ordering is easy to assert.
Pin metadata such as creation time and cwd, or candidate ordering drifts across machines.
DeepSeek Harness: Four assertion types
Four kinds cover success, failure, and lifecycle. (source) Checklist:
- Shape —
contentis still a block array with mention tokens removed; - Counts —
originalMessages = retainedMessages + omittedMessages, withomittedBytesabove zero; - Codes — failure cases compare
SESSION_REFERENCE_*, never copy; - No publish — cancellation asserts
additionalContextis absent.
Type two is cheap and catches bugs sharply: break the recording logic and the three numbers immediately disagree.
Type three must compare the code field, not a serialized error string.
Type four has to be written separately: many suites test "an error was returned" but never "no context was published".
DeepSeek Harness: A case table
Seven cases cover most of the seven codes plus the main flow. (source) Comparison:
| Case | Input | Assertion |
|---|---|---|
| Single reference | short | Exactly one additionalContext |
| Several references | short × 3 | Still at most one aggregated context |
| Zero references | empty references | No context, content unchanged |
| Self reference | current session id | SELF_REFERENCE |
| Over the limit | more references than allowed | TOO_MANY |
| Over budget | long | truncated true with consistent counts |
| Cancellation | signal triggered | CANCELLED with no context |
The third row is the easiest to omit: zero references should produce neither an empty context nor an error.
Rows four and five are input validation and trigger with minimal data, no environment faults required.
DeepSeek Harness: Stable cancellation tests
Trigger the signal at a fixed point; do not race with sleep. (source) Approach:
- Provide a read implementation that can pause at a chosen stage;
- Abort while prepare is running;
- Assert
SESSION_REFERENCE_CANCELLED; - Assert that no
additionalContextis published afterward.
Step 4 is the point: cancellation means "do not publish", not merely "return an error".
If the implementation contains "write the context after storage finishes", this case fails immediately, which is exactly its value.
Never build the race with a fixed setTimeout: it passes intermittently on slow machines and fails intermittently on fast ones.
What not to do
Five practices make tests look green while proving nothing. Checklist:
- Asserting full prompt text — one wording change turns it red;
- Mocking
prepareitself — you test the mock, not the chain; - Depending on real session ids — they vanish in another environment;
- Judging truncation by wall-clock time — results vary by machine;
- Testing only the success path — none of the seven codes is covered.
The second is the most dangerous: coverage looks impressive while every regression slips through.
The first and fourth are the same mistake in different clothes: putting something nondeterministic into an assertion.
DeepSeek Harness: CI rollout order
Run three essentials first, then add boundaries. (source) Suggested order:
- Single reference plus zero references, covering the main flow and empty input;
- Over-budget truncation, covering counts and
truncated; - Self reference, over-limit, and cancellation as failure cases.
All three groups run without a key and finish quickly, so they suit every commit.
Once fixtures and assertions are stable, add branches for source compaction and format versions, so too many variables do not arrive at once.
When tests are stable, return to the fields themselves. Read the DSH plugin reference budget and 64 KiB fallback; to design assertions by error code, read the seven DSH plugin reference error codes; for plugin testing overall, read how to test a DSH plugin.
Sources: DeepSeek Harness reference - Session references, session-reference/src/index.ts
FAQ
No, DSH plugin tests can run without a real model. Without model capacity or an adapter, the automatic budget is 64 KiB, a deterministic value that suits stable assertions in a no-model environment.
Text changes with implementation and wording, so DSH plugin tests should assert stable contracts instead. DSH plugin's stable contract is the field shape, count relationships, and error codes.
Trigger the DSH plugin signal at a fixed point rather than relying on timing. DSH plugin publishes no context after cancellation, so assert that additionalContext is absent without waiting on real delays.
No, DSH plugin fixtures do not need real conversation content. Fixtures only need the session structure and a few identifiable strings; DSH plugin assertions target shape and counts, not dialogue text.
Related Terms
- 64 KiB fallback
- The 64 KiB fallback is the fixed automatic budget used when model capacity or an adapter is missing, making preparation predictable without a model.— DeepSeek Harness reference - Session references
- fixed fixture
- A fixed fixture is a prebuilt session with stable content so reference results can be compared repeatably.— session-reference/src/index.ts
- shape assertion
- A shape assertion checks return structure and types, such as content being a block array and at most one additionalContext.— DeepSeek Harness reference - Session references
- no-publish assertion
- A no-publish assertion verifies that cancellation placed no additionalContext into the message.— DeepSeek Harness reference - Session references
Sources
- DeepSeek Harness reference - Session references· deepseek-ai
- session-reference/src/index.ts· deepseek-ai