DeepSeek Harness subprocess U+FFFD: fix truncated UTF-8

DeepSeek Harness truncation should remove content, never rewrite what remains; this implementation rewrites it. When subprocess output exceeds the collector's retention cap, the head of the buffer is dropped by byte count without regard for character boundaries, so the text the model receives begins with U+FFFD characters the command never printed. According to the discussion, the reporter measured this on a published package, and community members then reviewed output.ts in subprocess-local (discussion).

Symptom: a small DeepSeek Harness output cap cuts out two replacement characters

Whenever the cut lands inside a multi-byte sequence the text gains characters, and everything after them is correct, which makes the result look trustworthy. Reproduce with the script from the discussion:

import { OutputCollector } from '@deepseek-ai/dsh-subprocess-local/lib/output.js'

const c = new OutputCollector(23, 'stdout', undefined)
c.push(Buffer.from('编译完成：3 个文件 ✅ done\n', 'utf8'))

console.log(JSON.stringify(c.finalize().text))
// "��3 个文件 ✅ done\n"   two characters the input never contained

console.log(JSON.stringify(c.readFrom(0).text))
// "��3 个文件 ✅ done\n"   the foreground read is corrupted too

Check the following in order so unrelated problems do not get mixed in:

1. Set the cap well below the output size. Expected: truncated is true.
2. Print finalize().text. Expected: one or two U+FFFD characters at the start.
3. Print readFrom(0).text. Expected: the same corruption, so this is not specific to one read path.
4. Find the spill file from the same run. Expected: byte-exact content with no replacement characters.

The corruption happens in the DeepSeek Harness buffer, not in the decoder

The actual operation is this.chunks[0] = head.subarray(excess): a raw byte slice, with no UTF-8 boundary handling anywhere in the file. Every later call to finalize(), snapshot(), or readFrom(0) decodes bytes that were already broken. The community reviewed output.ts:167-181:

while (this.bytes > this.maxBytes) {
  const head = this.chunks[0] as Buffer
  const excess = this.bytes - this.maxBytes
  if (head.length <= excess) {
    this.bytes -= head.length
    // drop the whole chunk
  } else {
    this.chunks[0] = head.subarray(excess)
    this.bytes -= excess
  }
}

Three consequences follow:

1. The damage happens during buffering, so every downstream reader inherits it, including the foreground read on the main path.
2. Fixing the decoder alone is not enough, because the bytes reaching it are already missing their lead byte.
3. truncated: true cannot distinguish "dropped the head cleanly on a character boundary" from "dropped the head mid-character". Both cases share one value.

Fix 1: advance the DeepSeek Harness cut to the next UTF-8 lead byte

This is the smallest change with the most stable semantics: keep the byte budget and move the cut forward to a character start, dropping at most three extra bytes. Steps:

1. Replace the raw slice with a trim function that takes the original buffer and the number of bytes to drop. Expected: the trim logic has one entry point instead of a bare slice.
2. Walk forward from excess while the current byte satisfies (byte & 0xC0) === 0x80. Expected: the offset lands on a lead byte or the end of the buffer.
3. Call subarray with the new offset and write the number of bytes actually dropped back to this.bytes. Expected: this.bytes reflects the bytes actually dropped.
4. Handle the edge case: if the walk reaches the end of the buffer, the whole chunk is undecodable, so drop it entirely. Expected: an all-continuation chunk is discarded rather than decoded.
5. Rerun the script from the first section. Expected: the opening Chinese text is intact and truncated is still true.

Illustrative implementation (not the repository's original code):

let cut = excess
while (cut < head.length && (head[cut] & 0xc0) === 0x80) cut++
this.chunks[0] = head.subarray(cut)
this.bytes -= cut

Fix 2: keep a byte-exact window in DeepSeek Harness and decode at read time

If downstream must receive exactly maxBytes bytes, do not repair the buffer; let each reader handle an incomplete sequence with a streaming decoder. Steps:

1. Keep subarray(excess) unchanged and record explicitly that this buffer may start with continuation bytes. Expected: the buffer layout is unchanged and the possible continuation prefix is documented.
2. Replace toString('utf8') in finalize(), snapshot(), and readFrom() with a shared TextDecoder in streaming mode. Expected: every read path uses the same decoder configuration.
3. Either skip to the first lead byte once at the start of the window, or mark and drop the incomplete sequence before decoding. Expected: decoding starts at a valid lead byte.
4. Rerun all three read paths. Expected: all three agree and none contains U+FFFD.
5. Confirm the spill path was not changed with them; it writes raw chunks and does not need decoding. Expected: spill output remains byte-exact and is unaffected. Switching DSH plugins or terminals does not change this trim either.

State the trade-off clearly: Fix 1 guarantees that the stored buffer is valid text, while Fix 2 guarantees that the window is exactly maxBytes. Without knowing the encoding in advance, you cannot have both. The adjacent spill short-write defect is covered in spill short write.

Fix 3: raise the DeepSeek Harness cap so truncation stops happening

Raising the cap only lowers the probability; it is a stopgap, not a resolution. Steps:

1. Find the current cap for the output collector and confirm it is a hard limit measured in bytes. Expected: the byte limit and its unit are confirmed.
2. Raise it to cover the output distribution of your real commands. Expected: ordinary build logs stop setting truncated.
3. Test once with output that clearly exceeds the new cap. Expected: replacement characters still appear, which proves this is only mitigation.
4. At the same time, state in the result that the omitted amount is computed from the bytes actually returned, so the cap and the returned value do not disagree. Expected: the omitted count matches the bytes actually returned.

Verify the DeepSeek Harness fix: three text classes and two controls

Acceptance cannot rely on one Chinese sentence; it must cover 2-, 3-, and 4-byte code points and include a case that must fail and one that must pass. Steps:

1. Prepare a sample mixing Chinese, Latin letters, and emoji, iterate over several caps, and count how often U+FFFD appears at the start. Expected after the fix: zero.
2. Keep a pure-ASCII sample. Expected: no replacement characters at any cap.
3. Keep a sample known to cut mid-character. Expected before the fix: it hits 100% of the time; after the fix: zero. This proves the detector itself works.
4. Compare the spill file with the in-memory text. Expected after the fix: both start the same way, and truncated still means only that content was omitted.

Source: Discussion #10345.