ChatGPTClaudeGrok / X / CursorGemini

DeepSeek Harness Linux: dsh-subprocess Scope Unit Leak

TroubleshootingPublished 2026-11-03Author: DeepSeek Plugin Market
DeepSeek Harnesssystemd scopeTasksCurrentinotifysystemctlLinux
On Linux, dsh web leaves hundreds of dsh-subprocess scope units and a systemctl storm at 60% of one core, as release is gated on terminationRequested.

After DeepSeek Harness (DSH) has run dsh web for a long time on Linux and driven external agent teams, the scene can look like a system infection: systemctl holds more than 900 stale dsh-subprocess-* units, the process tree carries roughly 500 concurrent systemctl children, about 60% of one core burns inside child_process.spawn, and heap memory grows along with the handles. The root cause is neither systemd nor your kernel. It is SystemdScopeOwner.emptyRange() in @deepseek-ai/dsh-subprocess-local, which requires terminationRequested to be true before a scope is considered finished — and that flag is only set by signal() / terminate(), never by a command that completes normally. So the unit stays active with TasksCurrent=0 forever and gets polled by systemctl --user show every 5 seconds (#11635).

Three commands that show the scene: unit count, systemctl concurrency, inotify usage

The decisive signal is: a large number of stale units, TasksCurrent=0, a comparable number of systemctl --user show children, and an immediate CPU drop once they are cleared. Three diagnostic commands:

  1. Count the stale scope units.
bash
systemctl --user list-units --type=scope --state=active --plain --no-legend \
  | awk '{print $1}' | grep -E 'dsh-(subprocess|terminal)-'

Expected: a long list of dsh-subprocess-*.scope; the reported environment had more than 900. If the number is in the single digits, stop here — this is not the leak. 2. Count the concurrent polling systemctl children.

bash
ps -eo pid,ppid,etimes,cmd | grep 'systemctl --user show' | grep -v grep | wc -l

Expected: a count of the same order as the stale units, peaking around 500. Every stale owner polls every 5 seconds, and the interval doubles up to SYSTEMCTL_TIMEOUT_MS, but it never stops. 3. Check whether inotify watches are exhausted.

bash
journalctl --user -b | grep 'inotify watch descriptor'

Expected: a line like dsh-subprocess-leak1.scope: Failed to add control inotify watch descriptor ...: No space left on device. systemd learns "the scope is empty" only through an inotify watch on the cgroup's cgroup.events, and v255 merely logs the failed watch creation without compensating, so that scope is never reclaimed. 4. Clear one stale unit and watch the CPU. Run systemctl --user stop <unit>.scope for any unit with TasksCurrent=0. Expected: the next poll finds the unit gone or inactive, waitForExit() returns, the handle leaves the live set, and CPU drops immediately. The reporter's wording: "clearing the stale units collapsed the storm instantly".

The mechanism: the release condition is gated on "was terminated"

In one sentence: emptyRange() needs two conditions, and the second only holds for an active termination, so a normally finished command can never reach the cleanup branch. Unpacked:

  1. The first condition is correct: the scope is empty (TasksCurrent=0) and either the direct client has left (it already forked everything it was going to fork) or the start request has been consumed (the payload entered the scope).
  2. The second condition is wrong: it additionally requires terminationRequested. That field is set only by signal() / terminate(), covering the "kill / terminate" path; normal completion and startup failure never set it.
  3. So the cleanup path is never entered: releaseEmptyRange is never called, the unit stays active, and the waitForExit() observation loop keeps polling every 5 seconds.
  4. Nor is the handle released: only when waitForExit() ends does the handle leave the subprocess service's live set; while the loop runs, memory stays pinned.
  5. Commit background: aaa02a3970 introduced emptyRange / releaseEmptyRange and gated them on terminationRequested at the same time. The class's own comment already states that a departed client cannot add a payload and a consumed request proves the payload entered the scope — termination signalling and scope release are orthogonal concerns.

So this is not a "systemd version compatibility issue"; upstream tied two independent concepts together.

Deterministic reproduction: shrink the inotify watch budget to the used value

In one sentence: the leak needs systemd to "miss" the populated→empty transition, and exhausting the watch budget reliably produces that miss. Steps (systemd 255.4 / Ubuntu 24.04):

  1. Prepare a payload that exits after one second. Start a scope with systemd-run --user --scope and have the payload sleep one second, then exit.
  2. Shrink fs.inotify.max_user_watches to the amount uid 1000 already uses. Expected: the user manager reports No space left on device when adding the control inotify watch.
  3. Observe the unit state after the payload exits.
bash
systemctl --user show --property=LoadState --property=ActiveState --property=TasksCurrent dsh-subprocess-leak1.scope

Expected: ActiveState=active, TasksCurrent=0, and the load state still present — exactly the shape in the report. 4. Run an undisturbed control. Restore a sufficient watch budget and repeat step 1. Expected: the scope is reclaimed as soon as the payload exits, LoadState=not-found. The value of this reproduction is turning the trigger from a guess into an actionable item: any machine whose log contains Failed to add control inotify watch descriptor may hit the leak; a machine without that line needs another explanation even if it has many units.

Temporary stopgap: stop only stale units with TasksCurrent=0

In one sentence: enumerate the active scopes and run stop only on empty ones, leaving live scopes untouched, with no need to restart DSH. The command comes from the thread, copied line by line:

bash
systemctl --user list-units --type=scope --state=active --plain --no-legend \
  'dsh-subprocess-*' 'dsh-terminal-*' | awk '{print $1}' | while read -r unit; do
  if [ "$(systemctl --user show --property=TasksCurrent --value "$unit")" = 0 ]; then
    systemctl --user stop "$unit"
  fi
done

Section by section:

  1. The enumeration targets active scopes only, with globs matching just dsh-subprocess-* and dsh-terminal-*, so no other system scope is touched.
  2. TasksCurrent=0 is the one safety valve: stopping an empty scope has no side effects, while any scope still running tasks is skipped.
  3. The loop converges naturally after the stop: the next poll finds the unit gone or inactive, waitForExit() ends, the handle is released.
  4. Automate only via a user timer, never via reset-failed or a manager restart: wiring the loop to a timer every few minutes caps the unit count at a small ongoing cost, whereas systemctl --user reset-failed does not handle active units and a user-manager restart interrupts the tasks you are running.

Long-term mitigation and the real fix

In one sentence: on the user side you can raise fs.inotify.max_user_watches and clean periodically; on the product side the correct fix is to remove the terminationRequested gate. Two tracks:

User side:

  1. Check usage against the limit, then raise it and persist the value: compare cat /proc/sys/fs/inotify/max_user_watches with the count already in use, then write a larger value through sysctl. Expected: newly created scopes no longer fail to add a watch.
  2. Also find out who is consuming the watches: if the consumer is not DSH, raising the limit only postpones the problem and you must deal with the real occupant.

Product side (the patch proposed in the thread):

  1. Remove the terminationRequested gate from emptyRange() and delete the write-only field, so the remaining predicate alone proves the scope has conclusively ended and future maintainers are not misled.
  2. Add regression tests: a new case for "releases an active-empty scope after the client finishes, with no termination request", and rename the old case to "keeps waiting on an active-empty scope while the client is alive", correcting the waiting invariant from "no termination request" to "client alive".
  3. Verify at scale: the full subprocess-local suite passes 382 with 0 failures, covering terminal scope, spawn and windows-job suites.
  4. Fix status: as of that discussion it is unfixed; the same gate exists from 0.2.1-rc.3 through master ce7073d5. External PRs are not accepted, so report via Discussions per the contribution guide.

One easily missed detail: a normal command should not hit this at all

In one sentence: when systemd-run --scope runs normally, the payload exit is itself the populated→empty transition and systemd reclaims the scope; the leak requires that transition to be missed. The responder explicitly corrected the overgeneralization that "every normal finish leaks":

  1. Normal path: systemd-run --scope puts itself into the scope, bootstrap execve runs the payload, and the moment the payload exits is the populated→empty transition, at which systemd should stop and reclaim the unit.
  2. Leak path: systemd misses that transition (typically because the inotify watch could not be added), and only then does the unit sit at active + TasksCurrent=0.
  3. Read the two pieces of evidence separately: a watch-failure line in journalctl means the trigger is present; no such line but leftover units means look elsewhere and do not force this explanation.
  4. The safety of the patch: a starting command still holds the request file and a live client, and background children keep TasksCurrent above 0, so removing the gate cannot kill a running scope.

Debugging notes

For this "slowly gets worse over time" family, the rule is: prove resources are accumulating with system-level counters first, then go back to the code and find the release condition. Five points:

  1. Count units and child processes first; matching orders of magnitude mean a polling storm, not merely a slow query.
  2. TasksCurrent=0 is the safety precondition for cleanup; never run stop on a scope that still has tasks.
  3. inotify is a trigger, not the root cause: raising the limit only reduces new leaks, and clearing old units still has to be done separately.
  4. Do not "fix" it by restarting the host, and do not expect a version bump to help: stopping the stale units is enough, a restart interrupts running tasks, and the gate exists in rc.3, rc.5, alpha.2 and master alike.
  5. Distinguish dsh-subprocess-* from dsh-terminal-*: both are managed by the subprocess service, but terminal scopes may hold a session a user is actively interacting with, so cleanup must likewise go by TasksCurrent=0.

Source: Discussion #11635, related discussion #9220.

ChatGPTClaudeGrok / X / CursorGemini

FAQ

DeepSeek Harness Linux: are dsh-subprocess units eating CPU?

First count the stale DeepSeek Harness units and the systemctl child processes. Run systemctl --user list-units --type=scope --state=active --plain, filter for dsh-subprocess-*, then count concurrent systemctl --user show processes with ps. If the two counts are the same order of magnitude and a single core is busy, you are almost certainly looking at a polling storm from stale scopes.

Why does a DeepSeek Harness scope persist at TasksCurrent=0?

Because DeepSeek Harness's emptyRange() not only requires the scope to be empty but also requires terminationRequested to be true, and only signal() or terminate() sets that flag. A command that finishes on its own never issues a termination request, so the cleanup path is never entered. systemd only reclaims a scope on the populated→empty transition, and once that transition is missed the unit stays active indefinitely.

Reproduce DeepSeek Harness stale scopes on systemd 255?

To reproduce it in DeepSeek Harness, shrink fs.inotify.max_user_watches to the value already in use by uid 1000, then run a payload through systemd-run --user --scope that exits on its own after one second. Expected: the user manager log shows Failed to add control inotify watch descriptor ... No space left on device, and after the payload exits the unit is still active running with TasksCurrent=0.

Clear leaked DeepSeek Harness scopes without a restart?

Yes, DeepSeek Harness leaks these units but they can be cleared safely. Enumerate the active scopes matching dsh-subprocess-* and dsh-terminal-*, and run systemctl --user stop only for those whose TasksCurrent=0, leaving live scopes alone. After the stop, the next poll finds the unit gone or inactive, the loop ends, the handle is released, and CPU drops immediately.

Related Terms

systemd scope
A systemd scope is a unit type that does not predefine a start command; systemd-run --scope puts the current process into that scope and runs the payload directly. A scope is reclaimed only when its cgroup goes from populated to empty, so missing that transition leaves the unit active for a long time.— https://github.com/deepseek-ai/deepseek-harness/discussions/11635
TasksCurrent
TasksCurrent is the attribute systemd exposes for the current number of tasks in a cgroup, readable with systemctl show --property=TasksCurrent --value. A value of 0 on a still-active unit usually means the scope is empty but systemd has not reclaimed it.— https://github.com/deepseek-ai/deepseek-harness/discussions/11635
terminationRequested
terminationRequested is an internal flag in DeepSeek Harness' subprocess scope, set only by signal() / terminate() to mean the caller actively asked to terminate. Treating it as a necessary condition for scope release makes every normally finished command skip the cleanup path.— https://github.com/deepseek-ai/deepseek-harness/discussions/11635

Sources