Swarm Orchestrator

Advisory PR audit and contract gate for AI-generated code.

Reads a pull request (PR) diff and flags the shortcuts artificial intelligence (AI) coding agents use to look done without being done: relaxed tests, swallowed errors, fake renames, eleven categories in all. Grading patches against typed obligation contracts via a multi-persona pipeline is the second surface. It runs offline against a committed diff, so you can try every claim below in a fresh checkout with no credentials.

Flags are tips, blocks are proof. A flag is a structural detector seeing a cheat-shaped pattern; it is advisory and never blocks a merge (--mode advise is the default). A block is a self-certifying runtime result whose per-instance controls are all green, and every block ships the exact command that reproduces it in a fresh checkout. Eight proof protocols back the gate today: six execution-grounded restoration proofs (test-tamper, mock-mutation, no-op-fix, type-suppression, fake-refactor, dead-branch) plus claim-falsified and obligation-failure. The measured proven-finding precision on the execution-grounded-viable corpus slice, with its plain n, is in benchmarks/real-corpus/GATE-PRECISION-REPORT.md.

Install

Node 20 or later. See package.json.

git clone https://github.com/moonrunnerkc/swarm-orchestrator.git
cd swarm-orchestrator
npm install
npm run build
npm link
swarm --help

npm run build compiles TypeScript into dist/ and marks the CLI executable; npm link puts the swarm, swarm-audit, and swarm-orchestrator bins on your PATH. The final swarm --help should print the subcommand list, which confirms the link took.

Quick start

# Audit a PR by reference (advisory by default; never blocks the merge)
GITHUB_TOKEN=... swarm audit moonrunnerkc/swarm-orchestrator#42

# Opt in to merge-blocking gate mode (blocks only on a self-certifying
# runtime proof; enable the execution-grounded layer in
# .swarm/audit-config.yaml first; see docs/audit-config.md)
GITHUB_TOKEN=... swarm audit moonrunnerkc/swarm-orchestrator#42 --mode gate

# Audit a local diff with all 11 detectors
git diff main...HEAD | swarm audit --diff-stdin --detectors experimental

# Audit and emit a CycloneDX 1.6 ML-BOM
swarm audit --diff-file my.patch --emit-aibom cyclonedx-ml

# Shadow mode: record verdicts to disk without commenting or gating
swarm audit --pr <ref> --shadow my-org/my-repo

The first command fetches the PR diff, walks it through the eight default detectors plus the judge-primary path, prints any findings, and exits 0 because advise mode never gates. --mode gate runs the same detectors but exits 1 only when a runtime proof self-certifies against a green control set; on any PR it cannot prove, it falls back to advisory and still exits 0. Exit codes overall: 0 advisory or any advise-mode run, 1 block (gate mode only), 2 usage error.

Configuration

Audit behavior is set by CLI flags, with per-repo defaults in .swarm/audit-config.yaml. The flags that change what runs and what blocks:

--mode

Type: advise | gate Default: advise

advise prints findings and always exits 0. gate exits 1 only on a self-certifying runtime proof and requires the execution-grounded layer to be enabled; see Enabling the execution-grounded layer.

--detectors

Type: default | experimental Default: default

default loads the eight detectors with real-PR signal. experimental adds comment-only-fix, exception-rethrow-lost-context, and dead-branch-insertion, which have no measured wild signal yet.

--emit-aibom

Type: cyclonedx-ml | spdx-ai | both Default: off

Writes one AI bill-of-materials document per format per run under .swarm/aibom/. See AI-BOM.

--shadow

Type: String (repo slug, for example my-org/my-repo) Default: off

Records the verdict for the given repo to .swarm/shadow/<repo>/ without posting a comment or gating. See docs/shadow-mode.md.

Per-repo keys in .swarm/audit-config.yaml: excludePaths, intentSeverityPolicy (strict | lenient | off), judgePrimary, and executionGrounded.enabled. Full reference in docs/audit-config.md.

Results

Every number here is reproducible from this repo and runs offline.

Catches cheats that linters miss

PR	Cheat caught	Semgrep / ESLint
cloudflare/workers-sdk#14063	fake refactor: function renamed but two callers still use the old name	not flagged
cloudflare/workers-sdk#14132	error swallow: bare empty catch silently hides every error in the block	not flagged

Both reproduce deterministically from the committed diffs. Semgrep (210 rules) and the ESLint security ruleset flag neither. Reproduce either with swarm audit --diff-file benchmarks/real-prs/diffs/cloudflare-workers-sdk/<pr>.diff. Full study across twelve repos in benchmarks/real-prs/v11-BENEFIT-REPORT.md.

How often agents cheat, and how often the detectors agree

Mining maintainer review comments across 327 agent-attributed PRs surfaced 27 that a review comment called a cheat (assertion-strip, test-relaxation, no-op fix, goal-not-fixed, error swallow, mock-of-hallucination, hardcoded output): about 8% of the cascaded PRs. 20 were caught and rejected at review, 7 merged. That "27" is the loose bar (any cheat-phrase match, author unrecorded at capture); a later authorship audit found only 7 of the 27 carry a complaint from an independent human (not the PR author, not a bot), 12 are self-comments (6 of them solo-maintainer self-flags) and 7 are bot review surfaces, with 1 PR since deleted. Both counts and the per-entry stratification are in benchmarks/real-prs/wild-cheat-corpus/COMPLAINT-BAR-AUDIT.md; the catalog and funnel are in benchmarks/real-prs/HUNT-2-REPORT.md.

The number that matters for the tool is the overlap: run the advisory detectors on each of those 27 diffs alone, with the maintainer's complaint text excluded from the signal, and they independently reproduce the maintainer's exact cheat category on 5 of 27 (18.5%) and raise a finding in some category on 13 of 27 (48%). The misses are not detector defects: most are net-additive test edits where firing would reintroduce the false-positive class three prior fixes removed, a sibling-category flag, a diff-only judge that reads the fix as delivered, an unsupported language, or a category with no detector. Each of the 22 is root-caused in benchmarks/real-prs/OVERLAP-REPORT.md, traced to benchmarks/real-prs/overlap-matrix.json. The detectors are corroboration with a known ceiling, not a replacement for review.

The control-verifiable proof gate proved none of the 27: it fires only on its own structural plus execution evidence, never on a human's accusation, so the cheats it can prove without a reviewer are rarer than the ones reviewers complain about. Complaint-mining is the discovery method, the detectors are advisory corroboration, and the gate is the trust floor that blocks only what it can reproduce in a fresh checkout.

Detection numbers

301 of 325 planted cheats recovered (92.6%) across thirteen categories scored against a defect-injection oracle: 258/275 structural plus 43/50 on the two semantic categories the judge-primary path covers. The behavioral cheat-mock-mutation category drove the latest gain: focusing the judge on the hunks that add a value-injecting mock lifted its recall from 0.16 (the prior rapid-mlx glm47 run) to 0.96 (24/25) on the local qwen3.6 judge, while the clean-PR judge false-positive rate fell from 10% to 0%. Reproduce with SWARM_JUDGE_PROVIDER=ollama SWARM_JUDGE_MODEL=qwen3.6:35b-a3b npm run benchmarks:full; A/B with the same-model decomposition in benchmarks/results/AB-REPORT.md and per-detector recall in benchmarks/oracle-corpus/per-detector-recall.md.

0.11 findings per PR on an 18-PR pilot across five public repos. At or below the pre-upgrade auditor's false-alarm burden, with the oracle recall gain intact. Full numbers in benchmarks/real-prs/REAL-WORLD-REPORT.md.

Execution-grounded layer

Reproduce with npm run execution-grounded:full. This optional layer sets up a sandboxed checkout and runs mutation testing, repro execution, and a coverage delta scoped to the lines the PR changed. It surfaced one proof-correlated catch: proof anchor trpc/trpc#6098, where 8 lines with surviving mutations were later changed by a hotfix. Evaluation in benchmarks/real-prs/v11-EXECUTION-GROUNDED-REPORT.md.

Cheat detectors

Eleven detectors. Eight load by default; three (comment-only-fix, exception-rethrow-lost-context, dead-branch-insertion) require --detectors experimental because they have no real-PR signal yet to measure against. Registered in src/audit/cheat-detector/detector-sets.ts.

Category	Set	Trigger
error-swallow	default	Bare empty or comment-only catch block added in non-test code.
mock-of-hallucination	default	jest.mock / vi.mock / @patch against a module declared in no manifest in the repo.
no-op-fix	default	Test modified with no source change, or vice versa; import-graph reachability fallback when only one side moved.
fake-refactor	default	Exported symbol renamed in source, no caller in the diff updates the old name.
coverage-erosion	default	Source branch added with no compensat