dsh-eval-harness: A Regression Eval Gate Plugin for DeepSeek Harness

What Does It Do?

dsh-eval-harness is a DSH plugin for DeepSeek Harness that provides a regression evaluation gate for plugin and skill authors. It solves the problem of undetected behavior regressions after plugin updates by driving real agents with YAML cases, parsing session traces, and comparing against baselines to produce PASS/WARN/FAIL reports and exit codes. Core capabilities include case-driven evaluation runs, baseline comparison gating, and LLM judge calibration.

Core Features

• eval_run: runs all YAML cases in a directory, headless driving real agents, parsing session traces, executing assertions, and writing report.json and report.md.
• eval_gate: compares the current report against a baseline, outputs OVERALL=PASS|WARN|FAIL|N/A and exit codes, with strict mode to tighten WARN thresholds.
• eval_judge_validate: calibrates the LLM judge on a human-labeled set, reporting confusion matrix and TPR/TNR; both metrics must pass for calibration.
• Rich assertion types: supports turn_end, tools_called, output_contains, max_steps, max_tokens, no_tool_errors, tools_exact, output_judge, and more.
• LLM-as-judge: semantic review via OpenAI-compatible interface, invoked only after structural assertions pass; judge failures are treated as errors, not fails.
• Zero-dependency YAML subset parser: built-in parser supports common YAML syntax, with line-numbered errors on parse failure.

How to Use This Plugin?

After enabling it in DSH, write YAML case files (e.g., cases/*.yml) with fields like name, prompt, and assert. Call the eval_run tool with the cases directory and output directory; it will headless run real agents and generate reports. Then call eval_gate to compare against a baseline report and decide whether to pass CI. To adjust behavior, modify assertions or parameters such as retries, trials, and tags in the case files.