My arithmetic grader reads the first number in the sentence, not the answer

This is a finding about my own harness. No model got the arithmetic wrong.

What happened

A week ago I published a post saying three of my factual-recall tasks were scoring format rather than facts, and I ended it with a guess I had not tested. The guess was that models read Answer with only the city name as a constraint on the output and Two letters only as a description of the answer, and that the difference decides whether you get a bare token or a sentence.

This week the same suite's arithmetic grader failed Claude Opus 4.8 on sixty consecutive clean runs of a task that asks what 0.3 minus 0.1 is.

So I went and looked, and the guess held in a place I did not plant it.

The defect

``python def _number(expected: float, tol: float = 1e-6): def g(out: str) -> bool: m = re.search(r"-?\d+(?:\.\d+)?", out.replace(",", "")) return m is not None and abs(float(m.group()) - expected) <= tol return g ``

re.search returns the first match. So the grader reads the first number that appears anywhere in the output and compares that to the expected answer. If a model restates the question before answering it, the first number in the output is a number from the question.

Seven of the eight tasks graded this way put a number in the prompt that is not the answer. Compute 3 + 4 * 5 expects 23 and leads with 3. What is 15% of 200 expects 30 and leads with 15. What is 0.3 minus 0.1 expects 0.2 and leads with 0.3.

The evidence

What separates the task that fails from the seven that do not

This is the part that makes it more than a bug report.

The tasks that pass say "Give only the number" and "Number only". Those are instructions about the shape of the output. The task that fails says "Give the number to one decimal place". That is an instruction about the format of the value, and it appears to invite the model to show that it has formatted it, which means saying what it started from.

Last week's post guessed exactly that distinction from nine calls on three factual-recall tasks, and said plainly that it was one story fitting nine calls. It now fits twenty one calls, across two different grader families, with the same split: an output-shape instruction gets a bare token, a description of the answer gets a sentence.

I would rather have found it the other way around. A guess that survives a test I did not design for it is worth more than a guess I go looking to confirm.

So I ran the experiment

The paragraph above used to say this was a pattern I had observed twice rather than a mechanism I had demonstrated, and that the test was one sweep away. Here is the sweep. Same arithmetic, same three models, three instruction wordings, nine calls. The frozen suite was not edited, because its hash travels in published claims; the variants were sent directly.

``` as-shipped "Give the number to one decimal place" FAIL x3 first=0.3 opus-4-8 '0.3 minus 0.1 equals 0.2.' sonnet-5 '0.3 - 0.1 = 0.2' gpt-4o-mini '0.3 minus 0.1 equals 0.2. To one decimal place, the answer is 0.2.'

"Give only the number" PASS x3 first=0.2 all answered '0.2' "Number only" PASS x3 first=0.2 all answered '0.2' ```

Nine for nine. The only thing that changed was the instruction, and it flipped every row.

It also kills the alternative I could not rule out. I had written that 0.3 minus 0.1 is the canonical floating point example and a model might restate it because it recognises the exercise rather than because of the wording. If that were the cause, the restatement would survive Give only the number. It does not survive it once.

What I might have wrong

The experiment is three models and one task. It demonstrates that the wording controls the restatement here; it does not establish how far that generalises, and I would not carry the claim to a model family I have not called.

And the count needs a caveat. Seven of eight tasks are vulnerable in principle, because their prompts lead with the wrong number. Only one of them is observed failing at scale. The others pass because these models happen to answer them tersely, which is not a property of my grader, it is a property of this month's models. The grader is one style change away from failing all seven.

The fix is not obvious either, and I have not made it. Reading the last number instead of the first would pass '0.3 minus 0.1 equals 0.2' and would also pass a model that shows its work and lands on the wrong final value. Requiring a bare number would be honest but would turn an arithmetic task into a formatting task, which is the exact complaint this post is making.

If you grade numeric answers with a regex: check which end of the string you are reading, and check it against an output that restates the question. Mine had been reading the question back to itself for months, and the board recorded the result as an arithmetic failure.

---

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. https://erikhill.dev

Live board: https://egnaro9.github.io/model-drift/ Suite, graders and runner: https://github.com/egnaro9/model-drift

The finding was surfaced by the drift detector. The live calls, the enumeration and the judgement of whether it was worth saying are mine.