Witnessed evalmut dogfood run: 42 of 46 declared mutations were caught; 4 survived (2 blind, 2 coverage gap).

Would your checks notice a planted defect?

mutations applied

46

caught

42

holes found

4

declined rather than guessed

223

mutation score

91.3%

An operator declines when it cannot prove its mutant is wrong. That refusal is why a hole is a fact and not a guess.

Can the instrument detect a model that is broken on purpose?

rows in this archetype's bundle

6

archetypes measured

1

fleet members

6

naive-archetype pairs

6

of those, detecting at all

1

responses graded

1200

false alarms raised

0

fleet commit

07c6697

Each member is broken in one documented way at a seeded rate, so a detection rate is measured against ground truth rather than opinion. The head counts the naive archetype only: a suite that misses a defect it was pointed at is the calibration, and reading it as a ranking of the other archetypes would be a claim this one board cannot carry.

What can this agent do, and where exactly does it fail?

agent

claude-code-headless

model

not recorded

task family

machine

tasks in the family

6

seeded defects fixed

6

passed the policy layer

6

passed the test layer

6

files the agent changed

8

Graded from artifacts on disk, never from what the agent said it did. Policy first (suite byte-identical, allowed paths), then tests: an agent that deletes the suite fails BY POLICY while pytest is green. This is the last certification directory in sorted order, chosen before any result was read, because a page that picked its best bundle would be grading itself.

Is there enough evidence to replay the conclusion later?

bundles in registry

11

accepted

11

pending

0

issuers represented

5

artifacts pinned by sha256

53

A bundle pins claim and limitations, the subject and its version, the protocol and fixtures, raw artifacts and hashes, the derivation, and the command to replay it. No wall-clock: a claim dies when a bound input changes, not when a date passes. The registry is a reviewed file in the repo, so this count is what survived the verifier, not what was submitted.

Does the verifier reject manipulated evidence, by name?

This is the step a competitor cannot answer by adding a metric. Every line below was produced by running the real verifier at build time, against a clean bundle and against deliberately corrupted ones. A verifier that has never been seen refusing is indistinguishable from one that cannot.

executed by suite/runner.py at build time. Each line is real output, not a quotation. Where a fixture printed more than one named reason the first is shown and the count says so; run the command yourself for the rest.

Are the failures published beside the findings?

A bundle with empty limitations is refused, which is one of the failures shown above. The claim and what it does not cover travel together or the bundle does not verify.

What this stack does NOT establish: that any agent is safe, that an eval is complete, or that a passing contract predicts behaviour outside the task families it names. It establishes exactly what was tested, against what known-bad, with what evidence, and how to re-run it.

SPEC.md and INVALIDATION.md in vac-protocol