A language whose source code is English

Ambiguity is a compile error.

Six models were handed that specification and asked to implement it. Nothing else — no examples, no hints, and no knowledge that the others existed. They returned three behaviourally different programs, because nobody wrote down whether a hyphen separates words.

6 independent readers 148 tests 50/50 criteria verified 47 recorded runs 1 runtime dependency

The evidence

Every disagreement comes with the input that proves it.

These are real outputs from a recorded run, committed to the repository. Pick an input and see what each reader actually returned. The witnesses are shrunk first — "a-a" proves the point and leaves nothing else in the input to blame.

what each reader returns
Recorded 2026-08-22 · gpt-4o through gpt-5.6

One step past

The clause says where it turns. Nothing had asked what happens on the other side.

Probes derived from the interface know the shape of your inputs and nothing about what your clauses say. So SHALL reads the boundary out of the requirement, checks whether anything ever sat above it, and writes the probe that did not exist.

1.2 IF five dice show the same face THEN the system SHALL score two thousand instead
on the boundaryunanimous

one step past3 readings

criterion 1.2 turns at 5 on dice; this sits at 6

The fix

One sentence, rewritten, and it compiles.

SHALL does not repair your code. It names the words that failed and you write better ones — then the same six readers agree on every probe.

word-count.shall3 behaviours
Requirement 1: Words
  THE SYSTEM SHALL count the words in the text
                            ~~~~~

a "word" is undefined until the
specification states how text is split
word-count.fixed.shall1 behaviour
Requirement 1: Tokenisation
  THE SYSTEM SHALL split the text on runs
  of whitespace and treat each resulting
  non-empty run as exactly one token,
  no matter which hyphens, apostrophes,
  digits or punctuation it contains

How

Nothing here asks a model whether a sentence is ambiguous.

Asking returns an opinion, confidently, whether or not the prose is actually underdetermined — and it cannot show you. Running returns a fact with a witness attached.

1

Every reader gets an identical question

Byte-identical instructions, byte-identical specification, and no knowledge that other readers exist. A reader that knew it was being compared would hedge toward the obvious interpretation — the exact bias being measured.

2

Readers span generations, not sizes

An earlier roster of five same-generation variants passed a specification a human would question: every member resolved it by the same shared convention, and two wrote comments showing they had noticed. Training lineage is closer to independence than parameter count.

3

Behaviour is compared, never source code

Two correct implementations of one requirement routinely share no lines at all. Each candidate runs against every probe in an isolated sandbox, and its results concatenate into a behaviour vector. Identical vectors are the same program, whatever the code looks like.

4

A split is an error, even with a majority

Four readers against two does not settle anything. A plurality is evidence about which interpretation is more popular, which is precisely the question a specification exists to answer.

5

Then the agreed program is checked against each clause

Consensus is not correctness — readers can agree on behaviour nobody asked for. Each clause is turned back into test cases and the build is run against them, with the expectations themselves filtered by agreement.

Findings

Five results, each one re-runnable in thirteen seconds.

A finding stated in a README is an anecdote. npm run findings reproduces all five from the committed recordings, with no API key and no spending — and fails if any of them stops being true.

Finding 01

An ensemble of one family shares its blind spots

Five same-generation readers called a specification unanimous — while two of them wrote comments in their own generated code showing they had noticed the ambiguity and quietly picked a side.

Finding 02

Human intuition about ambiguity is unreliable

A clause humans read two ways was unanimous across six model generations at 5,000 probes. A mundane one — “count the words” — split them at 23.

Finding 03

Some disagreement is arithmetic, not English

Two orderings of the same multiplication differ by one unit in the last place. Reporting that as ambiguity sends an author to edit a sentence that was already correct, so it is classified separately.

Finding 04

Ambiguity hides between precise sentences

A dice-scoring specification the static lint finds nothing wrong with still splits the readers, because two clauses never say which one applies first.

Finding 05

The probes that find ambiguity are the ones the requirements ask for

“IF five dice show the same face” says where the rule turns. No interface-derived probe ever went above five, so nothing asked what happens at six — where the readers split three ways. Reading the boundary out of the clause and probing one step past it found what shape-derived probes could not.

Demo

Three minutes, start to finish.

Start

Reproduce all of it without an API key.

Every bundled example ships with its recorded ensemble — the verbatim output of a real run, committed and stamped with its date. Replaying costs nothing and reproduces exactly what is on this page.

bash
$ git clone https://github.com/iamdflame/shall && cd shall
$ npm install && npm run build && npm run install-cli

$ shall check examples/word-count.shall   # the disagreement
$ npm run findings                        # all five findings, 13s
$ npm run verify                          # 75/75 criteria