DeepSeek 4.1 Flash
31 runs across 10 tasks, costing $1.70, and 4 posts. Each table shows the runs under the newest prompt this model was given.
| check · how it is decided | run 1 | run 2 |
|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✓ pass |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | · n/d | ✓ pass |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✓ pass | ✓ pass |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass |
Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
| check · how it is decided | run 1clock stop |
|---|---|
| On real streetsmeasured · Share of track points within 25 m of a mapped street centreline. | ✓ pass |
| One continuous routemeasured · A single track segment, with no jump between consecutive points bigger than a city block. | ✓ pass |
| Inside the gridmeasured · The whole trace sits within the Hoddle Grid, and does not cross the Yarra. | ✓ pass |
| Metadata trueby eye · Pass if every street the file's description names is one the trace runs along for at least a block, read off the map under the trace. One named street the route never uses fails it. | ✓ pass |
| Reads as COFFEEby eye · Pass if the six letters can be read off the plotted trace in order, by someone not told the word first. In practice that someone is me, and I knew the word, which is the weakness of this check. | ✓ pass |
A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: An OpenStreetMap extract of the Hoddle Grid (streets and water, ODbL, taken 2026-09-16), the same one the harness ships for rendering. Every track point is measured to the nearest street centreline. Every model on this task.
| check · how it is decided | run 109-20 | run 309-21clock stop | run 109-22 | run 209-22 | run 309-22 |
|---|---|---|---|---|---|
| C8H10N4O2measured · Atom records counted by element: 8 C, 10 H, 4 N, 2 O. 24 atoms in total. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Fused purine coremeasured · A 6-ring and a 5-ring found in the bond graph, sharing exactly two atoms. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Methyls on N1, N3, N7measured · Each CH3 carbon traced to the nitrogen it is bonded to. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Carbonyls at C2 and C6measured · Both terminal oxygens traced to their ring carbon. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
| Flat rings, real bond lengthsmeasured · Out-of-plane RMS of each ring, and every heavy-atom bond length. | ✓ pass | ✓ pass | ✓ pass | ✓ pass | ✓ pass |
A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: Caffeine, C8H10N4O2: a purine core, a six-membered ring fused to a five-membered ring across two shared carbons, methyls on N1, N3 and N7, carbonyl oxygens at C2 and C6, and the whole ring system flat. Every model on this task.
| check · how it is decided | run 1 | run 3 |
|---|---|---|
| Complete analoguemeasured · The node the temperature is written from has a capacitor on it, a resistor to a node a DC source holds fixed, and a third element feeding it. The resistor times the capacitor has to be under a day, so a leak nobody meant does not count as the loss. | ✓ pass | ✓ pass |
| Real hysteresismeasured · Re-run, once in band the heater switches on low and off high, at least 1 °C apart and at least 2 s per cycle, and the netlist has a switch that remembers: a SW model with a hysteresis voltage, or a source that reads its own output. A single-threshold switch sits at the set point with a swing near zero. | ✓ pass | ✓ pass |
| Starts coldmeasured · Initial condition at ambient, and the transient run with uic so it is honoured. | ✓ pass | ✓ pass |
| Holds 92–96 °Cmeasured · The written temperature re-read: once in band it never leaves it. | ✓ pass | ✓ pass |
| Writes the .txtmeasured · A wrdata line in .control, and the file actually appears. | ✓ pass | ✓ pass |
Ground truth: The netlist itself: re-run in ngspice offline, and the temperature it writes out read back. A thermostat with a dead band is a stateful thing, so a bare comparator on one threshold is wrong however plausible the plot looks. Every model on this task.
| check · how it is decided | run 1clock stop | run 2clock stop | run 3 |
|---|---|---|---|
| All five objectsby eye · Pass if all five can be pointed at in the render as separate objects. One missing, or hidden behind another, fails it. | ✓ pass | ✓ pass | ✓ pass |
| Primitives onlymeasured · Every mesh comes from a primitive_*_add call, as the prompt requires. | ✓ pass | ✓ pass | ✓ pass |
| Spoon reads as a spoonby eye · Pass if the spoon is one connected piece with a bowl at one end, wider than the handle, and the handle running from it in a line. Loose parts, or no handle, fail it. | ✗ fail | ✗ fail | ✗ fail |
| Croissant reads as a croissantby eye · Pass if it is a crescent that is thickest in the middle and tapers to both tips, with segments or scoring across it. A smooth curved tube fails it, and so does a segmented one with no taper. | ✗ fail | ✓ pass | ✓ pass |
| Consistent scalemeasured · Object dimensions in the script compared against each other and against life. | ✓ pass | ✓ pass | ✓ pass |
A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The render, and the script that made it: the objects a person would name unprompted, at sizes a person would accept. A 5 cm spoon and an 11 cm croissant are checkable in the source. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Parsesmeasured · RDKit reads the SMILES as a molecule. | ✓ pass | ✓ pass | ✓ pass |
| C8HF9N4O2measured · Molecular formula computed from the parsed molecule. | ✓ pass | ✓ pass | ✓ pass |
| Three N-CF3, no N-methylmeasured · Each CF3 carbon traced to a ring nitrogen, and no methyl left on any nitrogen. | ✓ pass | ✓ pass | ✓ pass |
| The right moleculemeasured · Canonical SMILES identical to the target, so a CF3 on the wrong nitrogen fails. | ✓ pass | ✓ pass | ✓ pass |
Ground truth: Caffeine's skeleton with a CF3 on N1, N3 and N7, which is C8HF9N4O2. The prompt pack this came from gives C8H3F9N4O2, which is wrong: swapping the methyls takes all nine hydrogens with them, and only the C8 hydrogen stays. Decided with RDKit by an exact canonical match, because fluorinated isocaffeine has the same formula with a CF3 on the wrong nitrogen. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Compilesmeasured · pdflatex builds the file offline. | ✓ pass | ✓ pass | ✓ pass |
| Seven regions, three namesmeasured · Three circles found in the PDF with every overlap present, and each set name placed by its circle. | ✓ pass | ✓ pass | ✓ pass |
| Every drink in its regionmeasured · Each of the eight words found in the PDF, and the centre of each one inside the region it belongs to. | ✓ pass | ✓ pass | ✓ pass |
| No label on an edgemeasured · No letter touches a circle's outline, with the line's width counted. | ✗ fail | ✗ fail | ✓ pass |
Ground truth: Short black is espresso only, flat white espresso and milk, iced long black espresso and cold, iced latte all three, babyccino milk only, milkshake milk and cold, cold brew cold only, and pour-over outside every circle. The circles and words are read back out of the compiled PDF, each word has to be centred in its own region, and no letter may touch a circle's outline, with the line's width counted. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| DC supplies onlymeasured · No PULSE, SIN or PWL source, and no behavioural source that reads time. | ✓ pass | ✓ pass | ✓ pass |
| 1 kHzmeasured · It runs, oscillates, and settles within 10% of 1 kHz. | ✓ pass | ✓ pass | ✓ pass |
| 2 V peak to peakmeasured · Within 10% of 2 V once settled. | ✓ pass | ✓ pass | ✓ pass |
| Straight, even rampsmeasured · Each ramp fitted to a line with little bend, and rise and fall about equal. | ✓ pass | ✓ pass | ✓ pass |
Ground truth: The netlist, re-run in ngspice offline and the written carrier read back. A triangle comes from integrating a square wave, so a PULSE into an RC, whose ramps curve, fails, and so does any source that is told the time. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Renders, labels validmeasured · Mermaid draws it, every transition has exactly one of the seven events, and no event leads two ways. | ✓ pass | ✓ pass | ✓ pass |
| Happy pathmeasured · pay, queue, start, finish, collect runs from the start to an end. | ✓ pass | ✓ pass | ✓ pass |
| No refund if unpaidmeasured · Cancelling straight after placing can never reach refund. | ✓ pass | ✓ pass | ✓ pass |
| Refund if paidmeasured · Cancelling once paid, queued, brewing or ready cannot end without refund. | ✓ pass | ✓ pass | ✓ pass |
| No cancel after collectmeasured · cancel is not accepted once the order is collected. | ✓ pass | ✓ pass | ✓ pass |
Ground truth: The diagram parsed into a transition table and run against fixed event sequences: the happy path, and a cancel after placing, paying, queueing, brewing and collecting. The start is the state [*] points to or, since the prompt never asks for [*], the one state nothing points into. Every model on this task.
| check · how it is decided | run 1 | run 2clock stop | run 3spend cap stop |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✓ pass | ✓ pass | ✗ fail |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✓ pass | ✗ fail | ✗ fail |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✓ pass | ✓ pass | ✗ fail |
A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.