I design multi-agent pipelines, then measure whether the orchestration earns its cost. Several of the rules below were tightened after something failed, and four of those failures are written up here, each with a source you can read.
How I run agents
Route by role. Judgment roles (planning, critique, evaluation) run on stronger models and execution runs on cheaper ones, so cost follows how hard the decision is. A model gate refuses to run a judgment role on an under-powered model.
Nothing closes without proof. A unit of work stays open until an automated check finds its evidence on disk. A report that the work is done is not evidence that it is.
Review from zero context. Consequential changes go to a separate model session that never saw the code being written. It gets the diff and the standard. It can flag; it cannot ship.
Agents never approve their own consequences. Commits and deploys wait for a person, and an approval counts only if it came from the operator's own input. An agent that types its own approval is refused.
Automate the clicks, never the gates. Handoffs between roles run on their own. Autonomy climbs a named ladder from L0 to L4; the harness runs at L2, and each higher level removes a click, never a gate.
Orchestration multiplies whatever the agents get wrong, so agent output stays unverified until a check a machine can run says otherwise, and a failure is made loud rather than quiet.
Measured, not assumed
I assumed a planner and a judge would improve the output, so I built a tool to measure it: three pipeline shapes, the same 20 coding tasks, real models, $0.99 in total. Scoring executes the generated code; no model grades anything.
Shape
Calls
Score
Cost
Latency
one drafter
1
95%
$0.031
2.2s
planner, then drafter
2
90%
$0.264
9.1s
planner, two drafters, judge
4
80%
$0.692
18.3s
Calls per task; cost is the total for all 20 tasks; latency is the mean per task.
The four-call panel beat the single drafter on 0 of 20 tasks, lost 3 and tied 17, at 22.1x the cost and 8.4x the latency. Nothing errored across 60 runs. On this run it scored lower and took longer.
The tool then refused to call the panel worse. Only 3 tasks separated the two, and even a clean sweep of 3 gives p = 0.25 at best. So the result is three separate statements rather than a ranking: no evidence the panel helps on this suite, a measured cost for trying, and too few tasks to settle the rest.
Then the same question on a real task: does an agent pipeline beat plain code at deciding whether installed software is exposed to a known-exploited vulnerability? On the first 22 inventory and CVE pairs, a plain matcher plus one keyword rule, tuned on that set, scored as well as the best agent pipeline at $0. What the agents found was one error in my own answer key and one contested label. So I pre-registered a held-out set and pushed its answer key before anything ran on it: 57 pairs from 30 CISA KEV vulnerabilities that NVD first published on or after 2026-07-01. Answers are scored by code against the frozen key.
Approach
Correct
To a person
Wrong
Missed exposures
Cost
code
44
0
13
1
$0
code + keywords
45
0
12
1
$0
agent
48
3
6
3
$13.36
agent + verify + skeptic
47
10
0
0
$33.34
Counts are pairs out of 57; cost is the total for all 57. Agent + verify scored the same as agent and is left out.
The pre-registered comparison is code + keywords against agent + verify + skeptic, and on correct answers it is a tie: 45 against 47, exact McNemar p = 0.69, which 57 pairs cannot separate. The difference is where the errors go. The code gave 12 wrong answers, each a firm verdict, and one of them missed an exposed item. The pipeline with a skeptic gave none and sent the 10 pairs it was unsure of to a person, for $33.34. The keyword rule caught 2 of the 12 conditional pairs it was written for.
The answer keys come from three blind labelers that are Claude models, like the agents, so their errors may be correlated. Every label and ruling is published in the repository.
A banner announcing the armed gate sat in the conversation. The model took the rule on as its own job, saying it would watch for the approval, and the gate looked enforced in a transcript where no command had run (3edde6c, c7203c2). The banner moved outside model context, and the gate is now verified by firing its handler directly with no model in the loop. A public demo fires the real handlers with a self-issued approval, marked as coming from an extension rather than the keyboard, and shows the push refused.
The tests checked the consumer, never the producer.
The approval-gate tests for a multi-agent app staged model output straight into app state, so they exercised the code that reads a proposal and never the code that writes one. The fix moved the fake down to the wire: the real SDK client runs unmodified and only the bytes are scripted. The recorded request bodies then exposed a context leak between agents, one agent's output never reaching the next agent's prompt, which no state-level test could see.
The gate could not see what it was gating.
write-up, the receipt gate on this site, incident August 2026
A pre-commit gate accepted a receipt for an earlier version of a file as proof for every later version, for four days. The fix binds each receipt to the staged blob. Six tests written for that fix all passed against a mutant that compared only the first four characters of the hash; a seventh test kills it.
Eight rounds of adversarial review before I trusted a tool.
Independent agents tried to break the tool's verdicts, each finding reproduced by a runnable script and triaged against the source. False findings on a well-formed suite reached zero by round six and stayed there through round eight. Every confirmed finding is pinned by a regression test in a 90-test suite, and two fixes proposed during review were rejected as circular.
Operating record
~200completed work-arcs
~190human-gated checkpoints
74zero-context critic reviews
13periodic self-evaluations
One development harness, spring to summer 2026, counted from its private on-disk archive, so the counts are self-reported, and described in the harness write-up. Two of its gates are ported to the open-source pi-gates; the harness prompts and failure specifics stay private.
What I would do on your team
Design agent pipelines in which every stage can fail loudly, with verification as a stage of its own rather than a step inside generation.
Measure whether an added agent pays for itself before it ships: cost, latency, and which tasks it actually changes.
Put a gate on anything irreversible, and prove each gate fires with no model in the loop.
Keep evidence on disk for every closed unit of work, so a reviewer can check the claim instead of trusting it.
Every number on this page comes from a public repository or article linked on it; the operating-record counts are self-reported from a private archive. github.com/egnaro9 · contact@erikhill.dev