dsh-benchmark: A DSH plugin for Deterministic Benchmark Evidence in DeepSeek Harness
dongsheng123132/dsh-benchmark
dsh-benchmark provides deterministic, reproducible benchmark evidence for DeepSeek Harness, evaluating tools and plugins with fixed cases and versioned scoring.
dsh-benchmark is a benchmark plugin built for DeepSeek Harness, turning benchmarking into a reproducible and verifiable engineering practice. By freezing target revisions, file fingerprints, execution parameters, and expected outputs in an explicit manifest, every run produces a content-addressed report verified by SHA-256, making regression comparison evidence-based. The plugin provides three tools—inspect, run, and compare—plus a standalone MCP server for manifest linting and address computation. It focuses on execution-layer evidence, not subjective LLM quality, ideal for DSH developers who need rigorous regression testing.
How to Install
dsh plugin --profile web add github:dongsheng123132/dsh-benchmark- Category
- Development & Operations
- Platform
- DSH Plugin
- Author
- dongsheng123132
- Distribution
- Plugin
dsh-benchmark Key Features
dsh-benchmark Repo Summary
What Does It Do?
dsh-benchmark is a DSH plugin for DeepSeek Harness that provides deterministic, reproducible benchmark evidence for DSH tools and plugins. It solves the problem of non-reproducible benchmark results by freezing explicit cases, versioned scoring, and content-addressed reports, letting you run benchmarks directly in the harness and produce comparable regression evidence. The plugin focuses on the execution-evidence layer, not subjective LLM quality, and is maintained by dongsheng123132 under the MIT license.
Core Features
- Deterministic evidence protocol: an explicit manifest freezes suite name, target revision, file fingerprints, executable, argv, expected exit code, and output hashes, ensuring reproducibility.
- Safety model: shell interpolation disabled, workspace-bounded paths, minimal child environment, secret-bearing fields rejected, and reports contain only hashes and measurements.
- Content-addressed reports: each run writes a content-addressed report verified by SHA-256, enabling baseline regression comparison.
- Three registered tools: inspect (metadata and fingerprints), run (execute fixed cases and write report), compare (compare current and baseline reports against manifest thresholds).
- Standalone MCP server: provides manifest linting and report address computation without executing commands or touching the filesystem.
How to Use This Plugin?
After enabling it in DSH, you can use the three registered tools: first inspect to check protocol metadata and fingerprints, then run to execute fixed cases and generate a content-addressed report, and finally compare to compare the current report with a baseline, using manifest thresholds to decide regression. The plugin also offers a CLI and MCP interface; the CLI supports subcommands like inspect, while MCP accepts bounded inline JSON for validation and address computation. Actual benchmark execution is available only through the DSH tool and CLI surfaces.
This page is an independent rewrite of the plugin's official README — for authoritative documentation and the latest changes, refer to the source: dongsheng123132/dsh-benchmark. The plugin is third-party code that runs on your machine once installed; inclusion does not imply endorsement — please review the source before installing.