Finding
Venn
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Compilesmeasured · pdflatex builds the file offline. | ✓ pass | ✓ pass | ✓ pass |
| Seven regions, three namesmeasured · Three circles found in the PDF with every overlap present, and each set name placed by its circle. | ✓ pass | ✓ pass | ✗ fail |
| Every drink in its regionmeasured · Each of the eight words found in the PDF, and the centre of each one inside the region it belongs to. | ✓ pass | ✓ pass | ✗ fail |
| No label on an edgemeasured · No letter touches a circle's outline, with the line's width counted. | ✓ pass | ✓ pass | ✗ fail |
Ground truth: Short black is espresso only, flat white espresso and milk, iced long black espresso and cold, iced latte all three, babyccino milk only, milkshake milk and cold, cold brew cold only, and pour-over outside every circle. The circles and words are read back out of the compiled PDF, each word has to be centred in its own region, and no letter may touch a circle's outline, with the line's width counted. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Compilesmeasured · pdflatex builds the file offline. | ✓ pass | ✓ pass | ✓ pass |
| Seven regions, three namesmeasured · Three circles found in the PDF with every overlap present, and each set name placed by its circle. | ✓ pass | ✓ pass | ✓ pass |
| Every drink in its regionmeasured · Each of the eight words found in the PDF, and the centre of each one inside the region it belongs to. | ✓ pass | ✓ pass | ✓ pass |
| No label on an edgemeasured · No letter touches a circle's outline, with the line's width counted. | ✓ pass | ✓ pass | ✓ pass |
Ground truth: Short black is espresso only, flat white espresso and milk, iced long black espresso and cold, iced latte all three, babyccino milk only, milkshake milk and cold, cold brew cold only, and pour-over outside every circle. The circles and words are read back out of the compiled PDF, each word has to be centred in its own region, and no letter may touch a circle's outline, with the line's width counted. Every model on this task.
French press
The Opus runs are from 23 September, on the same prompt.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✓ pass | ✗ fail |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | ✓ pass | ✓ pass | ✗ fail |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✓ pass | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
| check · how it is decided | run 109-23 | run 209-23 | run 309-23 | run 110-01 | run 210-01 |
|---|---|---|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | ✗ fail | ✓ pass | ✗ fail | ✗ fail | ✓ pass |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✓ pass | ✓ pass | ✗ fail | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✗ fail | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
Shrub
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✓ pass | ✗ fail | ✓ pass |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✗ fail | ✗ fail | ✓ pass |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✓ pass | ✓ pass | ✓ pass |
Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✗ fail | ✗ fail | ✗ fail |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✓ pass | ✓ pass | ✗ fail |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✓ pass | ✓ pass | ✓ pass |
Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.
Cost
| Sonnet 5.5 | Opus 5.5 | |
|---|---|---|
| Venn | $0.0597 a run, $0.0896 a pass | $0.0807 a run, $0.0807 a pass |
| French press | $0.0486 a run, $0.0730 a pass | $0.2136 a run, no pass |
| Shrub | $0.0870 a run, $0.2609 a pass | $0.1062 a run, no pass |
Sonnet has equivalent results to Opus and is cheaper.
The 18 runs cost $1.7876. The 18 runs cost $1.79.