oh-my-knowledge: An Evaluation and Observability DSH plugin for DeepSeek Harness
lizhiyao/oh-my-knowledge
OMK provides evidence-backed evaluation and observability for prompts, RAG, skills, and agents in DeepSeek Harness, enabling A/B comparison and release decisions.
oh-my-knowledge is an evaluation and observability DSH plugin built for DeepSeek Harness, making every knowledge change evidence-backed. It runs controlled A/B evaluations comparing knowledge artifacts under the same model, outputting a one-line verdict, confidence interval, and failed samples. It also visualizes task trajectories in Studio and uses evidence-gated promotion to decide if a version is shippable. By capturing gaps from real usage and turning them into eval samples, it keeps evaluation close to production. Ideal for developers who need rigorous validation of prompts, RAG, skills, and agents.
How to Install
dsh plugin --profile web add oh-my-knowledge- Category
- Data & Knowledge
- Platform
- DSH Plugin
- Author
- lizhiyao
- Distribution
- Plugin
oh-my-knowledge Key Features
oh-my-knowledge Repo Summary
What Does It Do?
OMK is a DSH plugin for DeepSeek Harness that provides evidence-backed evaluation and observability for prompts, RAG, skills, agents, and workflows. It solves the problem of validating knowledge changes in AI applications, letting you observe real-world performance, measure version differences, and decide whether a change is effective and ready to ship. Core capabilities include controlled A/B evaluation, task trajectory tracing, evidence-gated promotion, and turning real usage into evaluation samples.
Core Features
- Controlled A/B evaluation: same model, same evaluation samples, only the knowledge artifact changes; outputs a one-line verdict, confidence interval, failed samples, and cost.
- Task trajectory visualization: open persisted DSH task trajectories in Studio with four lanes showing request, actions, results, and knowledge.
- Evidence-gated promotion: use
omk promote/omk evolveto accept a version or generate a better candidate based on evidence. - Learn from real usage:
omk observecaptures production gaps for review, which can become eval samples. - Preflight checks:
omk doctorvalidates artifact structure, dependencies, safety, and measurability.
How to Use This Plugin?
After enabling OMK in DSH, you can reuse the current profile for controlled evaluations. Quick start: run omk init demo to scaffold a demo project, then execute omk eval --control code-review-v1 --treatment code-review-v2 --dry-run to preview calls and cost, and finally run omk eval to generate an HTML report. To inspect a single Codex task, run omk studio to open the local conversation overview. The first run has only 3 samples, so the verdict is usually UNDERPOWERED—that's normal; grow to ~20+ cases before trusting a ship/no-ship call.
This page is an independent rewrite of the plugin's official README — for authoritative documentation and the latest changes, refer to the source: lizhiyao/oh-my-knowledge. The plugin is third-party code that runs on your machine once installed; inclusion does not imply endorsement — please review the source before installing.