Finding
I tested many models to see whether I could get statistical significance where a model knows the order of a filter stack but cannot draw it. For most models I tested we cannot say much, but the Opus 5.5 runs somewhat suggest that it knows the order but often cannot draw it in order.
| check · how it is decided | run 109-23 | run 209-23 | run 309-23 | run 110-01 | run 210-01 |
|---|---|---|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | ✗ fail | ✓ pass | ✗ fail | ✗ fail | ✓ pass |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✓ pass | ✓ pass | ✗ fail | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✗ fail | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
All seven models, as passed of decided on the order:
| In words | Drawn | |
|---|---|---|
| Opus 5.5 | 5 of 5 | 2 of 5 |
| Sonnet 5.5 | 5 of 5 | 2 of 3 |
| GPT-5.6 Sol | 5 of 5 | 2 of 3 |
| DeepSeek 4.1 Flash | 3 of 3, all after searching the web | 1 of 1 |
| GLM 5.3 FlashX | 0 of 4 | 0 of 2 |
| Qwen3.8 27B | 0 of 3 | 0 of 2 |
| Pareto | 0 of 2 | 2 of 3 |
Verdict
With p at 0.17, the Opus 5.5 seems to indicate that, in general, it knows the order but cannot draw it in order, which is interesting.
n = 5 runs an arm for Opus, 2 to 5 decided for the others. p = 0.17 for Opus, and 0.37 for all seven without the web runs, Fisher exact, committed before the runs.
The 35 runs in words cost $0.82, and the five Opus drawings $1.03. The 40 runs cost $1.85.