An ensemble of one family shares its blind spots
Five same-generation readers called a specification unanimous — while two of them wrote comments in their own generated code showing they had noticed the ambiguity and quietly picked a side.
Six models were handed that specification and asked to implement it. Nothing else — no examples, no hints, and no knowledge that the others existed. They returned three behaviourally different programs, because nobody wrote down whether a hyphen separates words.
Every disagreement comes with the input that proves it.
These are real outputs from a recorded run, committed to the repository.
Pick an input and see what each reader actually returned. The witnesses are
shrunk first — "a-a" proves the point and leaves nothing else
in the input to blame.
The clause says where it turns. Nothing had asked what happens on the other side.
Probes derived from the interface know the shape of your inputs and nothing about what your clauses say. So SHALL reads the boundary out of the requirement, checks whether anything ever sat above it, and writes the probe that did not exist.
IF five dice show the same face THEN the system SHALL score two thousand instead
One sentence, rewritten, and it compiles.
SHALL does not repair your code. It names the words that failed and you write better ones — then the same six readers agree on every probe.
Requirement 1: Words THE SYSTEM SHALL count the words in the text ~~~~~ a "word" is undefined until the specification states how text is split
Requirement 1: Tokenisation THE SYSTEM SHALL split the text on runs of whitespace and treat each resulting non-empty run as exactly one token, no matter which hyphens, apostrophes, digits or punctuation it contains
Nothing here asks a model whether a sentence is ambiguous.
Asking returns an opinion, confidently, whether or not the prose is actually underdetermined — and it cannot show you. Running returns a fact with a witness attached.
Byte-identical instructions, byte-identical specification, and no knowledge that other readers exist. A reader that knew it was being compared would hedge toward the obvious interpretation — the exact bias being measured.
An earlier roster of five same-generation variants passed a specification a human would question: every member resolved it by the same shared convention, and two wrote comments showing they had noticed. Training lineage is closer to independence than parameter count.
Two correct implementations of one requirement routinely share no lines at all. Each candidate runs against every probe in an isolated sandbox, and its results concatenate into a behaviour vector. Identical vectors are the same program, whatever the code looks like.
Four readers against two does not settle anything. A plurality is evidence about which interpretation is more popular, which is precisely the question a specification exists to answer.
Consensus is not correctness — readers can agree on behaviour nobody asked for. Each clause is turned back into test cases and the build is run against them, with the expectations themselves filtered by agreement.
Five results, each one re-runnable in thirteen seconds.
A finding stated in a README is an anecdote. npm run findings
reproduces all five from the committed recordings, with no API key and no
spending — and fails if any of them stops being true.
Five same-generation readers called a specification unanimous — while two of them wrote comments in their own generated code showing they had noticed the ambiguity and quietly picked a side.
A clause humans read two ways was unanimous across six model generations at 5,000 probes. A mundane one — “count the words” — split them at 23.
Two orderings of the same multiplication differ by one unit in the last place. Reporting that as ambiguity sends an author to edit a sentence that was already correct, so it is classified separately.
A dice-scoring specification the static lint finds nothing wrong with still splits the readers, because two clauses never say which one applies first.
“IF five dice show the same face” says where the rule turns. No interface-derived probe ever went above five, so nothing asked what happens at six — where the readers split three ways. Reading the boundary out of the clause and probing one step past it found what shape-derived probes could not.
Three minutes, start to finish.
Reproduce all of it without an API key.
Every bundled example ships with its recorded ensemble — the verbatim output of a real run, committed and stamped with its date. Replaying costs nothing and reproduces exactly what is on this page.
$ git clone https://github.com/iamdflame/shall && cd shall $ npm install && npm run build && npm run install-cli $ shall check examples/word-count.shall # the disagreement $ npm run findings # all five findings, 13s $ npm run verify # 75/75 criteria