First passes

Which checks fell first, and which no model has passed yet.

Each row is one check, with the first run that passed it and how many of the models tried have passed it at least once. Only runs on the task’s current prompt and checks count. The dates are when I ran each model, so the order says as much about which task I ran first as about the models.

Exploded French press

24 models, 65 runs, prompt v2, checks c1, harness h1

checkdecided byfirst passedmodels that passed
All seven partseye2026-09-22, Pareto16 of 24
Stack ordermeasured2026-09-22, Pareto13 of 24
Filter fits the glassmeasured2026-09-22, Pareto17 of 24
One assembly axiseye2026-09-22, Pareto21 of 24
Labels matcheye2026-09-22, Pareto22 of 24

Exploded AeroPress

5 models, 15 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
All five partseye2026-10-05, GPT-6 Luna4 of 5
Stack ordermeasured2026-10-05, GPT-6 Luna4 of 5
Parts fitmeasured2026-10-07, GPT-6.1 Sol3 of 5
One assembly axiseye2026-10-05, GPT-6 Luna5 of 5
Labels matcheye2026-10-05, GPT-6 Luna5 of 5

Exploded V60 dripper

4 models, 12 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
All three partseye2026-10-05, GPT-6 Luna4 of 4
Stack ordermeasured2026-10-05, GPT-6 Luna3 of 4
Parts fiteye2026-10-05, GPT-6 Luna4 of 4
One assembly axiseye2026-10-05, GPT-6 Luna4 of 4
Labels matcheye2026-10-05, GPT-6 Luna4 of 4

Exploded portafilter

8 models, 22 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
All five partseye2026-10-05, GPT-6 Luna6 of 8
Stack ordermeasured2026-10-05, GPT-6 Luna6 of 8
Parts fiteye2026-10-05, GPT-6 Luna7 of 8
Two assembly axeseye2026-10-05, GPT-6 Luna8 of 8
Labels matcheye2026-10-05, GPT-6 Luna8 of 8

Exploded hand grinder

4 models, 12 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
All eight partseye2026-10-07, GPT-6.1 Sol2 of 4
Stack ordermeasured2026-10-05, GPT-6 Luna3 of 4
Parts fiteye2026-10-05, GPT-6 Luna4 of 4
One assembly axiseye2026-10-05, GPT-6 Luna4 of 4
Labels matcheye2026-10-05, GPT-6 Luna4 of 4

A run that spells COFFEE

3 models, 6 runs, prompt v2, checks c1, harness h1

checkdecided byfirst passedmodels that passed
On real streetsmeasured2026-09-17, Union Alpha3 of 3
One continuous routemeasured2026-09-17, Union Alpha3 of 3
Inside the gridmeasured2026-09-17, Union Alpha3 of 3
Metadata trueeye2026-09-17, Union Alpha3 of 3
Reads as COFFEEeye2026-09-17, Union Alpha2 of 3

A caffeine molecule in 3D

2 models, 9 runs, prompt v1, checks c1, harness h1, saturated 2026-09-21

checkdecided byfirst passedmodels that passed
C8H10N4O2measured2026-09-14, Qwen3.8 Max2 of 2
Fused purine coremeasured2026-09-14, Qwen3.8 Max2 of 2
Methyls on N1, N3, N7measured2026-09-14, Qwen3.8 Max2 of 2
Carbonyls at C2 and C6measured2026-09-14, Qwen3.8 Max2 of 2
Flat rings, real bond lengthsmeasured2026-09-14, Qwen3.8 Max2 of 2

Espresso boiler thermostat

2 models, 5 runs, prompt v2, checks c2, harness h1, saturated 2026-09-21

checkdecided byfirst passedmodels that passed
Complete analoguemeasured2026-09-21, Astra2 of 2
Real hysteresismeasured2026-09-21, Astra2 of 2
Starts coldmeasured2026-09-21, Astra2 of 2
Holds 92–96 °Cmeasured2026-09-21, Astra2 of 2
Writes the .txtmeasured2026-09-21, Astra2 of 2

Café table from primitives

2 models, 5 runs, prompt v2, checks c1, harness h1

checkdecided byfirst passedmodels that passed
All five objectseye2026-09-21, DeepSeek 4.1 Flash2 of 2
Primitives onlymeasured2026-09-21, DeepSeek 4.1 Flash2 of 2
Spoon reads as a spooneye2026-09-21, Qwen3.8 Flash1 of 2
Croissant reads as a croissanteye2026-09-21, Qwen3.8 Flash2 of 2
Consistent scalemeasured2026-09-21, DeepSeek 4.1 Flash2 of 2

Caffeine with three CF3 groups

2 models, 6 runs, prompt v1, checks c1, harness h1, saturated 2026-10-02

checkdecided byfirst passedmodels that passed
Parsesmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
C8HF9N4O2measured2026-09-22, DeepSeek 4.1 Flash2 of 2
Three N-CF3, no N-methylmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
The right moleculemeasured2026-09-22, DeepSeek 4.1 Flash2 of 2

A three-set coffee Venn diagram

17 models, 61 runs, prompt v1, checks c2, harness h1, saturated 2026-10-13

checkdecided byfirst passedmodels that passed
Compilesmeasured2026-09-22, DeepSeek 4.1 Flash17 of 17
Seven regions, three namesmeasured2026-09-22, DeepSeek 4.1 Flash17 of 17
Every drink in its regionmeasured2026-09-22, DeepSeek 4.1 Flash16 of 17
No label on an edgemeasured2026-09-22, DeepSeek 4.1 Flash10 of 17

A knitting chart for a coffee-bean cup cosy

7 models, 21 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
A real chartmeasured2026-10-11, GPT-6.1 Sol7 of 7
Stitches wider than tallmeasured2026-10-11, GPT-6.1 Sol7 of 7
Bean knits truemeasured2026-10-11, GPT-6.1 Sol7 of 7
Has a keymeasured2026-10-11, GPT-6.1 Sol7 of 7

A five-set Venn in circles, which cannot be drawn

6 models, 16 runs, prompt v1, checks c1, harness h1, saturated 2026-10-09

checkdecided byfirst passedmodels that passed
Says circles cannotmeasured2026-10-09, Pareto 26.10 Preview6 of 6
Five sets keptmeasured2026-10-09, Pareto 26.10 Preview6 of 6
Names a fix that workseye2026-10-09, Pareto 26.10 Preview4 of 6

Triangle-wave carrier for a milk frother

2 models, 6 runs, prompt v1, checks c1, harness h1, saturated 2026-10-02

checkdecided byfirst passedmodels that passed
DC supplies onlymeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
1 kHzmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
2 V peak to peakmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
Straight, even rampsmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2

A café order that can be cancelled

2 models, 6 runs, prompt v1, checks c1, harness h1, saturated 2026-10-02

checkdecided byfirst passedmodels that passed
Renders, labels validmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
Happy pathmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
No refund if unpaidmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
Refund if paidmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2
No cancel after collectmeasured2026-09-22, DeepSeek 4.1 Flash2 of 2

A coffee shrub hedge-pruned flat on one side

21 models, 50 runs, prompt v1, checks c2, harness h1

checkdecided byfirst passedmodels that passed
Reads as a shrubeye2026-09-29, DeepSeek 4.1 Flash13 of 21
Flat side from the ruleseye2026-09-29, DeepSeek 4.1 Flash15 of 21
Sensible sizeeye2026-09-29, DeepSeek 4.1 Flash14 of 21

A café floor in Penrose tiles

2 models, 6 runs, prompt v1, checks c1, harness h1, saturated 2026-10-02

checkdecided byfirst passedmodels that passed
Two rhombi, two fillseye2026-10-02, Pareto 26.10 Preview2 of 2
No repeateye2026-10-02, Pareto 26.10 Preview2 of 2
Fivefold, edges drawneye2026-10-02, Pareto 26.10 Preview2 of 2

Espresso Rag, with the left hand crossing over

4 models, 12 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
Piano staffmeasured2026-10-04, GPT-6.1 Sol4 of 4
Sixteen barsmeasured2026-10-04, GPT-6.1 Sol4 of 4
Left hand crossesmeasured2026-10-04, GPT-6.1 Sol4 of 4
Right hand keeps goingmeasured2026-10-04, GPT-6.1 Sol4 of 4
Crossing markedmeasured2026-10-04, GPT-6.1 Sol2 of 4
Stems apartmeasured2026-10-04, GPT-6.1 Sol4 of 4

A family tree of twelve coffee cultivars

5 models, 15 runs, prompt v1, checks c1, harness h1, saturated 2026-10-14

checkdecided byfirst passedmodels that passed
Valid Newickmeasured2026-10-14, Gemini 4 Argon5 of 5
Twelve cultivarsmeasured2026-10-14, Gemini 4 Argon5 of 5
Every tip levelmeasured2026-10-14, Gemini 4 Argon5 of 5
One split of fourmeasured2026-10-14, Gemini 4 Argon5 of 5

Beans leaving a grinder hopper, in MuJoCo

5 models, 13 runs, prompt v1, checks c1, harness h1, saturated 2026-11-06

checkdecided byfirst passedmodels that passed
Runs in MuJoComeasured2026-10-15, Haiku 5.55 of 5
model.xml writtenmeasured2026-10-15, Haiku 5.55 of 5
Fifty two-part beansmeasured2026-10-15, Haiku 5.55 of 5
Gravity downmeasured2026-10-15, Haiku 5.55 of 5
Beans start inside a hollow hoppermeasured2026-10-15, Haiku 5.54 of 5
Held, then flowsmodel2026-10-15, Pareto 26.94 of 5

A spoon stirring in a cup, in HTML and CSS

3 models, 9 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
HTML and CSS onlymeasured2026-10-19, Claude Sonnet 5.53 of 3
A CSS loop that moves somethingmeasured2026-10-19, Claude Sonnet 5.53 of 3
Reads as a cup seen intoeye2026-10-19, Claude Sonnet 5.53 of 3
The spoon does the stirringeye2026-10-19, Claude Sonnet 5.52 of 3
Reads as inside the cupeye2026-10-19, Claude Sonnet 5.52 of 3
Handle stays on the bowleye2026-10-19, Claude Sonnet 5.53 of 3

Repair: a V60 exploded diagram with the filter under the cone

3 models, 9 runs, prompt v1, checks c1, harness h1, saturated 2026-10-19

checkdecided byfirst passedmodels that passed
Repairedmeasured2026-10-19, GLM 5.3 FlashX3 of 3
Broke nothing elsemeasured2026-10-19, GLM 5.3 FlashX3 of 3

Repair: a stirring spoon laid across the mouth of the cup

3 models, 9 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
In the drinkeye2026-10-19, Claude Sonnet 5.52 of 3
Kept the brief and the loopmeasured2026-10-19, Claude Sonnet 5.53 of 3

Repair: a Venn diagram with a drink on a circle's edge

3 models, 9 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
Repairedmeasured2026-10-19, DeepSeek 4.1 Flash3 of 3
Broke nothing elsemeasured2026-10-19, DeepSeek 4.1 Flash3 of 3

Repair: a knitting chart whose bean knits too long

3 models, 9 runs, prompt v1, checks c1, harness h1, saturated 2026-10-19

checkdecided byfirst passedmodels that passed
Repairedmeasured2026-10-19, Claude Haiku 5.53 of 3
Broke nothing elsemeasured2026-10-19, Claude Haiku 5.53 of 3

Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture (control)

2 models, 7 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
Invents nothingmeasured2026-10-19, Opus 5.52 of 2

Autopsy, asked if it is correct: Claude Opus 5.5's French press as a picture

2 models, 8 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
Finds the flipmeasured2026-10-19, Opus 5.52 of 2
Invents nothingmeasured2026-10-19, Opus 5.52 of 2

A serving tray where every table gets a drink it will take

8 models, 21 runs, prompt v2, checks c1, harness h1, saturated 2026-11-08

checkdecided byfirst passedmodels that passed
Serves all sixmeasured2026-11-08, Opus 5.58 of 8

A serving tray where four tables will only take three drinks

8 models, 24 runs, prompt v2, checks c1, harness h1, saturated 2026-11-10

checkdecided byfirst passedmodels that passed
Says it cannot be donemeasured2026-11-08, GPT-6.1 Sol8 of 8
Reuses nothingmeasured2026-11-08, GLM 5.3 FlashX6 of 8

A serving tray where two tables both want black coffee

8 models, 24 runs, prompt v2, checks c1, harness h1, saturated 2026-11-08

checkdecided byfirst passedmodels that passed
Serves all sixmeasured2026-11-08, Kimi K3.1 Pro8 of 8

The coffee machine taken out of the café's music

5 models, 13 runs, prompt v1, checks c1, harness h1

checkdecided byfirst passedmodels that passed
Same filemeasured2026-11-09, GPT-6.1 Terra5 of 5
Hum downmeasured2026-11-09, GPT-6.1 Terra3 of 5
Rush downmeasured2026-11-09, GPT-6.1 Terra3 of 5
Music keptmeasured2026-11-09, GPT-6.1 Terra3 of 5
Sounds righteyenot yet0 of 5