What Is api-relay-audit? Local AI Relay Security Audit

api-relay-audit is a local security audit plugin in the DeepSeek Harness (DSH Plugin) ecosystem that runs before you trust a third-party AI API relay or LLM proxy, detecting prompt injection, model substitution, tool-call rewriting, and Web3 wallet risks, and producing a reviewable Markdown audit report. Based on the official README, this article covers what it is, its core features, installation and enabling, the /relay-audit and CLI usage, and common troubleshooting so you can audit a relay locally and reproducibly before sending production or wallet traffic.

What Is api-relay-audit?

api-relay-audit solves the "untrusted" problem of third-party AI API relays and LLM proxies: it runs 14 security probe steps locally against the relay you specify, detecting prompt injection, prompt extraction, instruction override, context truncation, tool-call rewriting, error leakage, and SSE stream anomalies, and outputs per-step findings with a final LOW / MEDIUM / HIGH verdict. The positioning and facts below all come from the official README (source):

api-relay-audit is maintained by toby-bridges and open-sourced under the AGPL-3.0 license, written in Python. It splits API relay audit, prompt injection audit, model substitution signals, and Web3 relay audit into separate query families so each result keeps a clean evidence boundary. The plugin runs locally: your API key is sent only to the relay URL you specify via --url, never to third-party services; the standalone audit.py is a single file that depends only on Python stdlib and curl. It does not certify relay safety, does not replace manual review or monitoring, and does not treat inconclusive as clean.

What Are the Core Features of api-relay-audit?

The core capability of api-relay-audit is local, repeatable, evidence-boundary-clean relay security auditing: it covers prompt injection, model substitution, tool-call rewriting, SSE anomalies, error leakage, and Web3 wallet risks, and every run produces a structured Markdown report with a LOW / MEDIUM / HIGH verdict. These capabilities all come from the official README (source):

• Detect relay tampering: covers token injection, prompt extraction, instruction override, jailbreak resistance, context truncation, tool-call substitution, error-response leakage, and SSE stream integrity.
• Collect model substitution signals: checks non-Claude identity leaks, anchor phrases, stream model identity, latency variance, and upstream channel fingerprints — these are signals that need corroboration before drawing conclusions.
• Audit tool-call rewriting: sends pinned package-install commands to the relay and compares the returned text to detect proxy-layer supply-chain tampering.
• Web3 wallet safety checks: with the web3 / full profile, checks transfer guidance, signed-transaction refusal, and private-key refusal behavior.
• Three runtime profiles: general (default, core probes), web3 (wallet-focused), and full (general plus Web3).
• Local reviewable evidence: the zero-dependency single-file audit.py outputs per-step findings plus a verdict, with an optional hash-only transparent log.

How to Install and Enable api-relay-audit?

Installing api-relay-audit requires pinning a release tag: a single dsh plugin command installs it into the current profile, and after restarting the gateway you can run /relay-audit in DSH; update means rerunning the install command with a new tag, and uninstall uses dsh plugin remove. The commands and facts below all come from the official README (source):

1. Install api-relay-audit: run the install command in the DeepSeek Harness terminal. The README requires pinning an immutable commit or release tag:

DSH_PLUGIN_REF=v2.4.0
dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"

Wait for the command to report a completed install.

2. Enable: restart the gateway and invoke /relay-audit: after install, restart the gateway:

dsh-restart

After the restart, type /relay-audit in DSH Web or a compatible TUI to run the audit; no arguments preserves the existing full-audit default, which may consume metered tokens.

3. Update the plugin: update means rerunning the install command with a new pinned release tag:

DSH_PLUGIN_REF=v2.4.0
dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"

Replace DSH_PLUGIN_REF with the latest tag and rerun to update.

4. Uninstall api-relay-audit: remove the plugin from the current profile:

dsh plugin --profile web remove dsh-api-relay-audit

Typical api-relay-audit Usage

Typical api-relay-audit usage has two entries: the /relay-audit slash command inside DSH audits the current provider route, while the standalone audit.py script audits any relay URL; both reuse the current DSH provider's baseURL, model, and credential reference. (source) The four steps below cover daily usage from a low-cost connectivity check to a Web3-specific audit.

1. Run a low-cost connectivity check: use --connectivity first to confirm the relay is reachable, avoiding the token cost of a full audit:

/relay-audit --connectivity

2. Run a Web3-specific audit: pass the web3 profile and fast-context when auditing wallet-related relay behavior:

/relay-audit --profile web3 --fast-context

3. Audit with explicit route parameters: specify the URL, model, and DSH credential explicitly:

/relay-audit --url <URL> --model <claude-model> --credential-ref <DSH_CREDENTIAL_REF>

4. Audit any relay with the standalone script: download the zero-dependency audit.py and run it directly; your API key goes only to the --url address:

python audit.py --key <YOUR_KEY> --url <BASE_URL> --profile web3 --output report.md

api-relay-audit Troubleshooting

The four most common api-relay-audit issues are Web3 checks not running, model identity signals mistaken for proof, inconclusive treated as safe, and high token consumption, fixed respectively with the web3/full profile, corroborating evidence, report reading, and --connectivity. (source)

1. Running general but no Web3 checks appear: symptom: the audit report shows no transfer guidance or signature refusal checks; cause: the Web3 relay audit is a separate query family, and only the web3 or full profile triggers the wallet-sensitive behavior checks. Fix: switch to the web3 / full profile:

/relay-audit --profile web3 --fast-context

2. Does a relay saying Qwen/DeepSeek prove model substitution: symptom: treating self-identification as proof of a model swap; cause: natural-language self-ID is a consistency signal, not upstream proof. Fix: combine raw response JSON, request IDs, provider/model metadata, and stream signatures as corroborating evidence before concluding.

3. Is an inconclusive result safe: symptom: unclear how to read a step whose probe was blocked or response was ambiguous; cause: blocked probes and ambiguous responses are not treated as clean. Fix: keep inconclusive results visible in the report, do not treat them as a safety certification, and combine them with manual review and monitoring.

4. Full audits consume too many tokens: symptom: high token consumption after running with no arguments; cause: no arguments preserves the full-audit default. Fix: run the low-cost check first:

/relay-audit --connectivity

Use Cases and Notes

api-relay-audit fits every scenario where you need to evaluate relay trustworthiness locally and reproducibly, but it does not certify relay safety, does not replace manual review or monitoring, and the audited route must identify as Claude. (source)

Use cases: using a third-party AI API relay, mirror, gateway, or LLM proxy; planning to use a relay for production traffic, coding-agent automation, package-install suggestions, or wallet-related actions; needing a local repeatable report instead of entering an API key into a web tool. Notes:

1. It does not issue safety certifications for any relay; the report is evidence, not a certificate, so combine it with manual security review and online monitoring.
2. inconclusive is never treated as clean — blocked or ambiguous probes stay visible in the report for you to interpret.
3. The plugin adds no new model baseline: the audited route must identify as Claude, although the relay API itself may be Anthropic-compatible or OpenAI-compatible.
4. The real API key is resolved only from DSH Credentials and delivered via a subprocess environment variable, never through command arguments or the session log; use redacted reports for public evidence submissions.

Project Links

api-relay-audit is an AGPL-3.0 open-source project maintained by toby-bridges. Plugin details: api-relay-audit plugin details.

This page is an independent guide rewritten from the plugin's official README — for authoritative documentation and the latest changes, please refer to the source: toby-bridges/api-relay-audit. A plugin is third-party code that runs on your machine at install time; inclusion is not an endorsement — review the source before installing.