DeepSeek Harness spill file loses bytes on a short write

The disk is not corrupting anything — DeepSeek Harness simply lets the write fall short and nobody checks, while the file stays published as complete output. Before suspecting the subprocess, note that the in-memory tail is usually perfect; only the fallback spill file is truncated (source).

Symptom: the DeepSeek Harness spill file is short while the memory tail looks right

Bottom line: the model reports a normal ending, but reading the full output file misses the tail — because those two outputs are not the same artifact.

1. Run a command with heavy output (a full build log, a cat of a large file) so it clearly exceeds the memory cap. Expected: the UI mentions a spill file you can read.
2. Compare the last lines in the memory tail with the last lines of the spill file. Expected: the endings match, but the file is shorter than it should be.
3. Measure the file with wc -c and compare it against the byte count of the command's real output. Expected: the file is small by an amount that is not a fixed truncation length.
4. Check the process log for write errors. Expected: nothing at all — this is the key signal, because a short write throws nothing.
5. If the size is right but the content contains replacement characters, that is a different defect; go to subprocess U+FFFD truncation instead.
6. Establish a baseline: redirect the same command to a local file once and record its byte count. Expected: every later comparison has a number to check against.

Root cause: two writeSync calls ignore their return values

Bottom line: spillAll() writes twice — replaying buffered chunks and writing the current chunk — and neither call compares bytes written against the buffer length.

According to community review of the source, writeSync(fd, prior) and writeSync(spillFd, chunk) are independent calls whose only guard is an outer try/catch. A short write neither throws nor changes control flow, so the incomplete file survives and keeps its published path (per the discussion).

Two details make the blast radius larger:

1. The spill path is assigned and published before either write happens, so callers already hold the path when the first short write occurs.
2. The only way to retract it is discardSpill(), and that is called only on a thrown write error or a stream-wide overflow. A short write hits neither.

Reproduce it with a 4-byte cap

Bottom line: you do not need a full disk to hit this — set the cap very low and make one write advance exactly 1 byte.

1. Take an instance where the output memory cap is configurable and set it to 4 bytes. Expected: any slightly larger output spills immediately.
2. Intercept writes on the collector's own file descriptor so the chosen call writes 1 byte and returns 1, leaving other I/O to Node's native implementation. Expected: the rest of the logic is unchanged.
3. Push AAAA, then BBBB, then CCCC. Expected: disk should end up with 12 bytes, AAAABBBBCCCC.
4. Read the file back. Expected: A + BBBB + CCCC, that is ABBBBCCCC; the 3 missing bytes are exactly what the replay-phase write dropped.
5. Move the interception point so the current chunk's write is the short one. Expected: a shorter result such as ABC, which shows both write paths can lose bytes independently.

Fix 1: loop writes in DeepSeek Harness until the buffer is drained

Bottom line: the correct primitive is a write loop with an offset — submit only the remaining slice each time and advance by the return value until the buffer is empty.

1. Change both writeSync(fd, buf) calls to the writeSync(fd, buf, offset, length) form and compute the remaining length first. Expected: the call signature matches Node's fs API.

let written = 0;
while (written < buf.length) {
  const n = writeSync(fd, buf, written, buf.length - written);
  if (n <= 0) throw new Error('short write made no progress');
  written += n;
}

2. Add the return value to the written count each round and continue from the new offset. Expected: a short write is completed by the next iteration.
3. Write the loop condition as "written < total", never "one call and done". Expected: no dependency on a single call filling the buffer.
4. Implement it for both writes, not just one; either the replay path or the current-chunk path can fall short. Expected: intercepting either path still produces a complete file.
5. Keep the original try/catch around the loop so real exceptions still take the failure branch. Expected: the exception path behaves exactly as before.
6. Re-run the 4-byte reproduction. Expected: 12 bytes on disk.

Fix 2: treat zero progress as failure and discard the spill in DeepSeek Harness

Bottom line: the loop alone is not enough — a writeSync returning 0 for a non-empty buffer would spin forever, so route that case to the existing failure exit.

1. Check the return value inside the loop and treat 0 as failure without retrying. Expected: no infinite loop.
2. On failure, call discardSpill() to retract the published path. Expected: callers see "no spill file" instead of a partial one.
3. Also fire the collector's existing failure callback so this output is marked incomplete. Expected: the log or status shows a failure rather than a silent success.
4. Confirm the failure branch reuses the same exit as the stream-overflow case. Expected: the "incomplete artifact does not count" contract has a single implementation.
5. Cover all three cases in regression tests: replay-phase short write, current-chunk short write, and a zero return. Expected: all three reach discardSpill.
6. After a failure, run the same command again. Expected: a new spill file is created rather than the old path being reused.

Fix 3: raise the DeepSeek Harness memory cap so spill never happens

Bottom line: if your output volume is predictable, the cheapest option is to stop spilling altogether — no spill file, no short-write path.

1. Measure the actual maximum output of these commands (take the p95 across several runs, not the single maximum). Expected: a defensible order of magnitude.
2. Set the subprocess output memory cap above that figure with headroom. Expected: ordinary commands stop producing spill files.
3. Re-run the same batch and confirm the UI no longer mentions a spill. Expected: memory use rises but stays acceptable.
4. Keep a monitor: any new spill means the cap is too low or the command changed. Expected: regressions show up early.
5. Be explicit that this is probabilistic relief — one overflow is enough for a short write to occur again, so it is a stopgap rather than an end state. No DSH plugin controls this loop; the collector belongs to the host.

Verify the DeepSeek Harness spill fix and choose a path

Bottom line: acceptance is numeric — the spill file's byte count must equal the full output's byte count, and a failure must not leave a published partial file.

1. Run all three paths against the 4-byte reproduction. Expected: the first two produce a complete file, the third produces none.
2. Run a real large-output command and reconcile with wc -c. Expected: identical byte counts.
3. Force a full disk or an exhausted quota. Expected: discardSpill runs and a readable failure signal remains.
4. Compare memory cost: Fixes 1 and 2 add almost nothing, Fix 3 adds a lot, so choose by machine spec.

Limits of this defect

Bottom line: this is a rare-condition bug, not a general weakness in file writes, and the loop, the zero-progress check and the discard path only work as one unit.

1. This is not a claim that ordinary writes often fall short — normal file writes usually complete in one call. The triggers are a full disk, quotas, interrupted writes, and partial completion on Windows.
2. Short writes and UTF-8 boundary truncation are two separate defects; do not fix them with one patch.
3. Any write loop needs a zero-progress guard, or you replace silent truncation with a CPU spin.
4. discardSpill() currently has only two call sites; when you add a failure exit, make sure it does not bypass them.
5. The locations and behaviour above come from community review of the source (baseline 0.2.1-rc.2); upstream has not fixed it, so re-confirm after an upgrade.

DeepSeek Harness FAQ

Bottom line: these are the questions that decide whether you are chasing this defect or a different output-collection problem.

Q: Can I detect this from the output alone? A: Only by byte count. The visible tail is correct, so the missing bytes are not visible at the end of the stream.

Q: Does the subprocess itself need to change? A: No. The truncation happens in the collector that persists the output, after the child has produced it.

Q: Is a short write always a disk problem? A: Usually a full disk or quota, but it can also come from an interrupted write or a partially completed write on Windows. The code must handle all of them.