DeepSeek Harness Linux: dsh-subprocess Scope Unit Leak
After DeepSeek Harness (DSH) has run dsh web for a long time on Linux and driven external agent teams, the scene can look like a system infection: systemctl holds more than 900 stale dsh-subprocess-* units, the process tree carries roughly 500 concurrent systemctl children, about 60% of one core burns inside child_process.spawn, and heap memory grows along with the handles. The root cause is neither systemd nor your kernel. It is SystemdScopeOwner.emptyRange() in @deepseek-ai/dsh-subprocess-local, which requires terminationRequested to be true before a scope is considered finished — and that flag is only set by signal() / terminate(), never by a command that completes normally. So the unit stays active with TasksCurrent=0 forever and gets polled by systemctl --user show every 5 seconds (#11635).
Three commands that show the scene: unit count, systemctl concurrency, inotify usage
The decisive signal is: a large number of stale units, TasksCurrent=0, a comparable number of systemctl --user show children, and an immediate CPU drop once they are cleared. Three diagnostic commands:
- Count the stale scope units.
systemctl --user list-units --type=scope --state=active --plain --no-legend \
| awk '{print $1}' | grep -E 'dsh-(subprocess|terminal)-'
Expected: a long list of dsh-subprocess-*.scope; the reported environment had more than 900. If the number is in the single digits, stop here — this is not the leak.
2. Count the concurrent polling systemctl children.
ps -eo pid,ppid,etimes,cmd | grep 'systemctl --user show' | grep -v grep | wc -l
Expected: a count of the same order as the stale units, peaking around 500. Every stale owner polls every 5 seconds, and the interval doubles up to SYSTEMCTL_TIMEOUT_MS, but it never stops.
3. Check whether inotify watches are exhausted.
journalctl --user -b | grep 'inotify watch descriptor'
Expected: a line like dsh-subprocess-leak1.scope: Failed to add control inotify watch descriptor ...: No space left on device. systemd learns "the scope is empty" only through an inotify watch on the cgroup's cgroup.events, and v255 merely logs the failed watch creation without compensating, so that scope is never reclaimed.
4. Clear one stale unit and watch the CPU. Run systemctl --user stop <unit>.scope for any unit with TasksCurrent=0. Expected: the next poll finds the unit gone or inactive, waitForExit() returns, the handle leaves the live set, and CPU drops immediately. The reporter's wording: "clearing the stale units collapsed the storm instantly".
The mechanism: the release condition is gated on "was terminated"
In one sentence: emptyRange() needs two conditions, and the second only holds for an active termination, so a normally finished command can never reach the cleanup branch. Unpacked:
- The first condition is correct: the scope is empty (
TasksCurrent=0) and either the direct client has left (it already forked everything it was going to fork) or the start request has been consumed (the payload entered the scope). - The second condition is wrong: it additionally requires
terminationRequested. That field is set only bysignal()/terminate(), covering the "kill / terminate" path; normal completion and startup failure never set it. - So the cleanup path is never entered:
releaseEmptyRangeis never called, the unit staysactive, and thewaitForExit()observation loop keeps polling every 5 seconds. - Nor is the handle released: only when
waitForExit()ends does the handle leave the subprocess service'sliveset; while the loop runs, memory stays pinned. - Commit background:
aaa02a3970introducedemptyRange/releaseEmptyRangeand gated them onterminationRequestedat the same time. The class's own comment already states that a departed client cannot add a payload and a consumed request proves the payload entered the scope — termination signalling and scope release are orthogonal concerns.
So this is not a "systemd version compatibility issue"; upstream tied two independent concepts together.
Deterministic reproduction: shrink the inotify watch budget to the used value
In one sentence: the leak needs systemd to "miss" the populated→empty transition, and exhausting the watch budget reliably produces that miss. Steps (systemd 255.4 / Ubuntu 24.04):
- Prepare a payload that exits after one second. Start a scope with
systemd-run --user --scopeand have the payload sleep one second, then exit. - Shrink
fs.inotify.max_user_watchesto the amount uid 1000 already uses. Expected: the user manager reportsNo space left on devicewhen adding the control inotify watch. - Observe the unit state after the payload exits.
systemctl --user show --property=LoadState --property=ActiveState --property=TasksCurrent dsh-subprocess-leak1.scope
Expected: ActiveState=active, TasksCurrent=0, and the load state still present — exactly the shape in the report.
4. Run an undisturbed control. Restore a sufficient watch budget and repeat step 1. Expected: the scope is reclaimed as soon as the payload exits, LoadState=not-found.
The value of this reproduction is turning the trigger from a guess into an actionable item: any machine whose log contains Failed to add control inotify watch descriptor may hit the leak; a machine without that line needs another explanation even if it has many units.
Temporary stopgap: stop only stale units with TasksCurrent=0
In one sentence: enumerate the active scopes and run stop only on empty ones, leaving live scopes untouched, with no need to restart DSH. The command comes from the thread, copied line by line:
systemctl --user list-units --type=scope --state=active --plain --no-legend \
'dsh-subprocess-*' 'dsh-terminal-*' | awk '{print $1}' | while read -r unit; do
if [ "$(systemctl --user show --property=TasksCurrent --value "$unit")" = 0 ]; then
systemctl --user stop "$unit"
fi
done
Section by section:
- The enumeration targets active scopes only, with globs matching just
dsh-subprocess-*anddsh-terminal-*, so no other system scope is touched. TasksCurrent=0is the one safety valve: stopping an empty scope has no side effects, while any scope still running tasks is skipped.- The loop converges naturally after the stop: the next poll finds the unit gone or
inactive,waitForExit()ends, the handle is released. - Automate only via a user timer, never via
reset-failedor a manager restart: wiring the loop to a timer every few minutes caps the unit count at a small ongoing cost, whereassystemctl --user reset-faileddoes not handle active units and a user-manager restart interrupts the tasks you are running.
Long-term mitigation and the real fix
In one sentence: on the user side you can raise fs.inotify.max_user_watches and clean periodically; on the product side the correct fix is to remove the terminationRequested gate. Two tracks:
User side:
- Check usage against the limit, then raise it and persist the value: compare
cat /proc/sys/fs/inotify/max_user_watcheswith the count already in use, then write a larger value through sysctl. Expected: newly created scopes no longer fail to add a watch. - Also find out who is consuming the watches: if the consumer is not DSH, raising the limit only postpones the problem and you must deal with the real occupant.
Product side (the patch proposed in the thread):
- Remove the
terminationRequestedgate fromemptyRange()and delete the write-only field, so the remaining predicate alone proves the scope has conclusively ended and future maintainers are not misled. - Add regression tests: a new case for "releases an active-empty scope after the client finishes, with no termination request", and rename the old case to "keeps waiting on an active-empty scope while the client is alive", correcting the waiting invariant from "no termination request" to "client alive".
- Verify at scale: the full
subprocess-localsuite passes 382 with 0 failures, covering terminal scope, spawn and windows-job suites. - Fix status: as of that discussion it is unfixed; the same gate exists from
0.2.1-rc.3through masterce7073d5. External PRs are not accepted, so report via Discussions per the contribution guide.
One easily missed detail: a normal command should not hit this at all
In one sentence: when systemd-run --scope runs normally, the payload exit is itself the populated→empty transition and systemd reclaims the scope; the leak requires that transition to be missed. The responder explicitly corrected the overgeneralization that "every normal finish leaks":
- Normal path:
systemd-run --scopeputs itself into the scope, bootstrapexecveruns the payload, and the moment the payload exits is the populated→empty transition, at which systemd should stop and reclaim the unit. - Leak path: systemd misses that transition (typically because the inotify watch could not be added), and only then does the unit sit at
active+TasksCurrent=0. - Read the two pieces of evidence separately: a watch-failure line in
journalctlmeans the trigger is present; no such line but leftover units means look elsewhere and do not force this explanation. - The safety of the patch: a starting command still holds the request file and a live client, and background children keep
TasksCurrentabove 0, so removing the gate cannot kill a running scope.
Debugging notes
For this "slowly gets worse over time" family, the rule is: prove resources are accumulating with system-level counters first, then go back to the code and find the release condition. Five points:
- Count units and child processes first; matching orders of magnitude mean a polling storm, not merely a slow query.
TasksCurrent=0is the safety precondition for cleanup; never runstopon a scope that still has tasks.- inotify is a trigger, not the root cause: raising the limit only reduces new leaks, and clearing old units still has to be done separately.
- Do not "fix" it by restarting the host, and do not expect a version bump to help: stopping the stale units is enough, a restart interrupts running tasks, and the gate exists in rc.3, rc.5, alpha.2 and master alike.
- Distinguish
dsh-subprocess-*fromdsh-terminal-*: both are managed by the subprocess service, but terminal scopes may hold a session a user is actively interacting with, so cleanup must likewise go byTasksCurrent=0.
Source: Discussion #11635, related discussion #9220.
FAQ
First count the stale DeepSeek Harness units and the systemctl child processes. Run systemctl --user list-units --type=scope --state=active --plain, filter for dsh-subprocess-*, then count concurrent systemctl --user show processes with ps. If the two counts are the same order of magnitude and a single core is busy, you are almost certainly looking at a polling storm from stale scopes.
Because DeepSeek Harness's emptyRange() not only requires the scope to be empty but also requires terminationRequested to be true, and only signal() or terminate() sets that flag. A command that finishes on its own never issues a termination request, so the cleanup path is never entered. systemd only reclaims a scope on the populated→empty transition, and once that transition is missed the unit stays active indefinitely.
To reproduce it in DeepSeek Harness, shrink fs.inotify.max_user_watches to the value already in use by uid 1000, then run a payload through systemd-run --user --scope that exits on its own after one second. Expected: the user manager log shows Failed to add control inotify watch descriptor ... No space left on device, and after the payload exits the unit is still active running with TasksCurrent=0.
Yes, DeepSeek Harness leaks these units but they can be cleared safely. Enumerate the active scopes matching dsh-subprocess-* and dsh-terminal-*, and run systemctl --user stop only for those whose TasksCurrent=0, leaving live scopes alone. After the stop, the next poll finds the unit gone or inactive, the loop ends, the handle is released, and CPU drops immediately.
Related Terms
- systemd scope
- A systemd scope is a unit type that does not predefine a start command; systemd-run --scope puts the current process into that scope and runs the payload directly. A scope is reclaimed only when its cgroup goes from populated to empty, so missing that transition leaves the unit active for a long time.— https://github.com/deepseek-ai/deepseek-harness/discussions/11635
- TasksCurrent
- TasksCurrent is the attribute systemd exposes for the current number of tasks in a cgroup, readable with systemctl show --property=TasksCurrent --value. A value of 0 on a still-active unit usually means the scope is empty but systemd has not reclaimed it.— https://github.com/deepseek-ai/deepseek-harness/discussions/11635
- terminationRequested
- terminationRequested is an internal flag in DeepSeek Harness' subprocess scope, set only by signal() / terminate() to mean the caller actively asked to terminate. Treating it as a necessary condition for scope release makes every normally finished command skip the cleanup path.— https://github.com/deepseek-ai/deepseek-harness/discussions/11635
Sources
- #11635 — bug(subprocess-local): an active-empty Linux scope unit is never released without a termination request· deepseek-ai (GitHub Discussions)