Fix RangeError in DeepSeek Harness session search

When a single session in a workspace grows large enough, session_search / session_event_search fail wholesale with Error: session query operation failed. The real cause is that search-index reconciliation serializes the entire event log into one string to compute a change fingerprint, crossing V8's single-string cap of about 512 MB (2^29−1 characters) and throwing RangeError: Invalid string length. Session listing and exact event reads keep working, so the failure stays hidden; the complete fix goes beyond an incremental fingerprint to incremental indexing, bounded snippets, serialized batch inspection, and CJK unigram+bigram tokenization.

DeepSeek Harness crash as it happens

The symptom looks like "search is broken", but what actually broke is an index-reconciliation chain unrelated to the query itself. What was measured:

1. Whole-workspace failure: as soon as any session in a workspace grows past the threshold, both session_search and session_event_search return Error: session query operation failed; the underlying RangeError: Invalid string length is thrown during search-index reconciliation, so search dies for the entire workspace, not just the big session (#1859).
2. It hides itself: session listing and exact event reads keep working, which makes the failure easy to miss; many people first assume only that one large session is affected.
3. Trigger size: the extreme case observed live had about 5 million events, while a persisted log of only about 150 MB raw / 223k events is already enough to cross the cap.
4. The real stack: RangeError: Invalid string length → at JSON.stringify → at observeSession (.../dsh-session-query-sqlite/lib/index.js:1000:49) → observeLive (:992) → Proxy._observeStable (:740) → async Proxy._reconcile (:651).
5. The error is downgraded: the tool-layer mapping (SESSION_QUERY_INVALID_CONFIG / SESSION_QUERY_SOURCE_CONFLICT in tool-session-query/service-boundary.ts) replaces the real exception with generic text, so debugging has to bypass it and read the host log.
6. Round two is worse: with only the incremental fingerprint applied, dsh web crashed instead with V8 heap exhaustion (about 4 GB, with constant Mark-Compact churn). It crashed once after roughly 15 hours, and once only about 5 minutes after a restart when a workspace search was triggered — a single session_search call was enough to kill the process (#1859).

DeepSeek Harness mechanism: whole-log stringify past the V8 cap, and zero CJK matches

Two independent mechanisms — one about a memory boundary, one about a tokenization boundary. Layer by layer:

1. One serialization of the whole log: observeSession() (packages/session-query/session-query-sqlite/src/index.ts) computes the change fingerprint with createHash('sha256').update(JSON.stringify({ header: detachedHeader, events: detachedEvents })).digest('base64url') — the entire log becomes one string.
2. Past V8's single-string cap: V8 caps one string at roughly 512 MB (2^29−1 characters); beyond that JSON.stringify throws RangeError, reconciliation fails, and the query aborts.
3. Heap pressure goes beyond that string: every search re-observes the entire corpus, so cost scales with the workspace rather than with the query. Four specific multipliers: (1) observeLive() clones every event of every live session and builds a per-event search document for each; (2) any session whose fingerprint changed — i.e. any active conversation — has all its FTS documents deleted and re-inserted on every search; (3) for a giant matched document the snippet path materializes the full highlight() output as a JS string (the Builtins_ArrayPrototypeJoin frame in the crash stack); (4) first-search indexing holds every inspected session's cloned events and documents in memory simultaneously.
4. Batch inspection concurrency is a killer too: readTitleSnapshots inspects persisted logs with a concurrency of 4 (SESSION_QUERY_DEFAULT_PERSISTED_INSPECT_CONCURRENCY = 4), holding up to four decompressed logs in memory at once. On a real corpus (the fanqie workspace, with sessions of 5.48M / 5.28M / 2.42M / 1.04M events) the title batch peaked at 2.36 GB heap / 4.06 GB RSS and was the "OOM 3 minutes 19 seconds after launch" culprit; this path fires on every @-mention autocomplete keystroke (session-reference) and after every session_search (title reads in tool-session-query) (#1859).
5. The exact-read paths clone whole logs too: SessionCorpus.load() cloned the entire log for every exact read (snapshotLive() mapped structuredClone over a 5.4M-event live snapshot, and the persisted branch cloned every event again). tracing.analyzeEventLog() additionally materialized one record object per event (events.map(event => ({...})), exactly the Builtins_CreateShallowObjectLiteral / ArrayMap crash signature) for readSurface (fired when selecting an @-reference) and for traceEvent.
6. Two independent reasons Chinese queries returned nothing: (1) the whole query was quoted as one FTS5 phrase, so multi-term queries required exact adjacency; (2) FTS5's unicode61 treats a run of consecutive CJK characters as one token, so searching 内存 inside 内存溢出, or any single character, returned zero hits.
7. Round three was a liveness deadlock: the v9 index reset marks every session as changed, so the first search re-inspects the entire corpus (about 12–15 seconds in that workspace); meanwhile the current chat session's write-behind flushes to its persisted log every few seconds. The stability gate compares snapshot revisions taken before and after the inspection window, and any churn retries the entire observation — when the rebuild window is longer than the flush interval it never stabilizes, both retries exhaust, every search fails, and the index stays empty forever. The decisive detail is that the churning revision belongs to a live session whose indexed rows are shadowed by the TEMP overlay and from which nothing is ever written by this path — so it should not have been watched at all.

DSH plugin patch branch and workarounds

The fix has four layers: bound the fingerprint, make observation and indexing incremental, bound the snippets, then fix the stability gate and CJK tokenization. Specifically:

1. Incremental fingerprint (the minimal change): hash the header once and each event separately, keeping memory bounded:

const fingerprintHash = createHash('sha256').update(JSON.stringify(detachedHeader))
for (const event of detachedEvents) {
  fingerprintHash.update('\n')
  fingerprintHash.update(JSON.stringify(event))
}

The side effect is that every session's fingerprint changes once, triggering a one-time index rebuild, which is expected whenever the hash construction changes. 2. Incremental observation and indexing: a per-Session WeakMap cache with an incremental SHA-256 stream and a cached surface fold; events are deep-frozen and the public snapshot array is replaced on append, so fingerprints hash only new events and no log content is ever cloned in the observation path. Only documents appended since the last indexed seq are inserted; positional replacements update only newly shadowed older rows (surface='shadowed', chunked IN lists); cache bookkeeping happens only after COMMIT; persisted sessions' inspect() results are no longer cloned, documents stream into the index at write time, and each session's log is dropped as soon as its rows are written. 3. Bounded snippets: window highlight() in SQL around the first match marker with instr / substr, so per-row JS memory is bounded by the snippet window; Array.from(text).length was replaced by an allocation-free code-point counter. 4. CJK substring search: CJK runs are stored as unigram+bigram token streams with zero-width separators, and queries expand their CJK runs into the same bigrams; whitespace-separated terms are ANDed as individually quoted literals so MATCH syntax stays inert data; snippets decode the token stream back to readable text. Schema v8→9 resets the derived index in place once (expected, since it is disposable by design). 5. Read paths stop cloning: persistedInspectConcurrency now defaults to 1 (a batch holds at most one inspected log in memory; small corpora can still raise it); SessionCorpus.load() no longer copies logs (live reads borrow the frozen snapshot array, persisted reads hand over the freshly inspected values, and callers clone only what they retain); tracing folds into lightweight relationship maps, and readSurface / traceEvent build only the single record they need. 6. Stability-gate fix: samePersistenceSnapshots now ignores snapshot entries owned by live sessions (taking ctx.sessions.list() ids at comparison time), while non-live churn (new sessions, deleted sessions, a detached log genuinely changing) still retries as before. 7. Install a tripwire first: dsh-plugin-doctor v1.10.0 adds a large-files check under --profile that warns when any profile file exceeds 100 MB (skipping node_modules/.pnpm), naming the relative path and size:

npx dsh-plugin-doctor --profile ~/.dsh/profiles/web --json

Note that the v9 index reset makes the first search after upgrading markedly heavier, which is exactly when this warning earns its keep. To get content out of a huge session, dsh-shelf's rescue exports it to markdown without loading the whole thing into one string, and verify flags unhealthy sessions. Install and update plugins from DSH Plugin Hub under Settings, Plugin Market. 8. Where the patch lives: the official repository was not accepting PRs at the time, so the complete fix (round one plus round two plus the stability fix) is committed to the fork branch flyingcoding/deepseek-harness @ fix/session-query-cjk-memory, commit c9abb534d, based on official 0.1.0-rc.7 (99f6f02fec) and ready to cherry-pick. One known remaining bound: readSession / listEvents still allocate their complete output by contract (they merely stop cloning the log first), and the UI transcript reader does not use them for rendering.

DSH plugin troubleshooting notes

Look past the generic message to the real exception first — session query operation failed usually hides a RangeError and an observeSession stack, and the failure has nothing to do with the query itself. Six points to keep in mind when a DeepSeek Harness plugin search breaks across a workspace:

1. Read the real exception first: besides the generic session query operation failed, look at the RangeError and the observeSession stack in the host log.
2. The threshold is lower than intuition suggests: about 150 MB raw / 223k events is enough; you do not need millions of events.
3. An incremental fingerprint is not the finish line: it fixes the string cap but not the heap pressure of re-observing the whole corpus on every search.
4. Live sessions prevent convergence: the stability gate must exclude snapshot entries owned by live sessions.
5. Chinese search needs bigrams: unicode61's single-token treatment of a CJK run will not change because of how you phrase the query.
6. The first search after upgrading is heavy: the schema reset rebuilds the derived index once, which is expected.

Sources: Discussion #1859, fix/session-query-cjk-memory, dsh-plugin-doctor v1.10.0.