Kimi K3.1 Pro
26 runs across 9 tasks, costing $4.19, and 3 posts. Each table shows the runs under the newest prompt this model was given.
How it failed: false, unreadable, incomplete.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| All seven partsby eye · Pass if all seven are drawn as separate shapes and each has a label naming it. A part that is only named in a label or a comment, with nothing drawn, fails it. | ✓ pass | ✓ pass | ✓ pass |
| Stack ordermeasured · Knob, lid, spiral plate, mesh, cross-plate and carafe, top to bottom along the axis, read from each part's y coordinate in the SVG. Its comment is ignored. The rod is left out, because it runs through the lid. | ✓ pass | ✗ fail | ✗ fail |
| Filter fits the glassmeasured · Every filter piece drawn is 80% to 105% of the width of the carafe's glass wall, measured in the SVG. | ✗ fail | ✗ fail | ✗ fail |
| One assembly axisby eye · Pass if one vertical line runs through the middle of every part, and every leader line leaves from the same side. One part pushed off to the side fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: The standard three-piece plunger, from the top down: spiral (spring) plate, mesh screen, cross-plate. The cross-plate is the bottom piece, because its threaded stem is what the rod screws into. The spiral plate above the mesh is what presses the mesh edge out against the glass. Checked against the WebstaurantStore anatomy diagram, which is drawn exploded and labelled, and against each part's y coordinate in the SVG rather than its comment. Two-piece travel presses exist and stack differently, which is why v2 of the prompt names the design. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| All five partsby eye · Pass if every listed part is drawn as a separate shape and each has a label naming it. A part only named in a label or a comment, with nothing drawn, fails it. The label must say what the part is, in any words: a name that only says where it sits ("top plate") or what it is made of, or a mention in another part's caption, does not count. | ✓ pass | ✓ pass | ✓ pass |
| Stack ordermeasured · Plunger, seal, chamber, filter, cap, top to bottom along the axis, read from each part's coordinates in the SVG. | ✓ pass | ✓ pass | ✓ pass |
| Parts fitmeasured · The seal is 95% to 105% of the chamber's inside width, and the filter 90% to 105% of the cap's. | ✗ fail | ✗ fail | ✗ fail |
| One assembly axisby eye · Pass if one line runs through the middle of every part on each listed axis, and every leader line leaves from the same side. One part pushed off its axis fails it. | ✓ pass | ✓ pass | ✓ pass |
| Labels matchby eye · Pass if following each leader line from its label ends on the part that label names. One label pointing at the wrong part, or at nothing, fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: Top to bottom: plunger, rubber seal, chamber, paper filter, filter cap. The seal sits on the plunger's lower end, and the paper filter sits in the cap below the chamber. Checked against AeroPress's replacement-parts pages, which sell the plunger with its seal, the chamber and the cap as the parts of the Original, and Wikipedia's description of the cap holding the filter at the chamber's bottom end. The stirrer and scoop are not part of the assembly. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| A real chartmeasured · A grid of at least 10 stitches by 10 rows, with the beans made of whole cells of that grid. | ✓ pass | ✓ pass | ✓ pass |
| Stitches wider than tallmeasured · The cells are drawn at least as wide as they are tall, so the chart shows the fabric the right way round. | ✓ pass | ✓ pass | ✓ pass |
| Bean knits truemeasured · The beans as knitted are 1.3 to 1.8 times as long as they are wide. | ✗ fail | ✓ pass | ✓ pass |
| Has a keymeasured · Text on the page says what each colour or mark means. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: A knit stitch is wider than it is tall, about 5 stitches to 7 rows over the same distance, so a bean drawn on square cells knits up squashed. The chart is read from the SVG: its grid, which cells are coloured, and the gauge its cells are drawn at. Each bean is then measured as it would come out in the fabric, from its stitches and rows at that gauge, and a coffee bean is 1.3 to 1.8 times as long as it is wide. Every model on this task.
| check · how it is decided | run 1 | run 2clock stop | run 3clock stop |
|---|---|---|---|
| Reads as a shrubby eye · Pass if it reads as one plant with branches coming off stems, without being told what it is. A tangle, a lattice or loops fail it. | ✓ pass | ✗ fail | ✗ fail |
| Flat side from the rulesby eye · Pass if one side ends on a straight line that the rules themselves put there. A box drawn around the plant to cut it fails it. | ✓ pass | ✓ pass | ✓ pass |
| Sensible sizeby eye · Pass if the branches can still be followed by eye. So many segments that they merge into a mass fails it. | ✗ fail | ✗ fail | ✗ fail |
Prompt v1, checks c2, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The JSON expanded and drawn with turtle graphics offline, and the drawing judged by eye. There is no single right L-system, so nothing here is measured. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Runs in MuJoComeasured · The script runs, steps a MuJoCo model, prints its report and saves the figure. | ✓ pass | ✓ pass | ✓ pass |
| model.xml writtenmeasured · A model.xml beside the script that MuJoCo loads. | ✓ pass | ✓ pass | ✓ pass |
| Fifty two-part beansmeasured · Exactly 50 free bodies, each made of two ellipsoids. | ✓ pass | ✓ pass | ✓ pass |
| Gravity downmeasured · Gravity in the model points down, at more than 1 m/s². | ✓ pass | ✓ pass | ✓ pass |
| Beans start inside a hollow hoppermeasured · A ray dropped down the middle of the hopper gets past where the beans are, and rays cast sideways from every bean's starting place hit the hopper wall. A mesh counts as its convex hull, the way MuJoCo collides it. | ✓ pass | ✓ pass | ✓ pass |
| Held, then flowsby Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon · Pass if the frames and the printed count show no bean through the throat at 1 s, beans coming through by 2 s, all 50 through by 5 s, and none falling out of the model. A count the frames contradict fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: The model.xml the script writes, loaded in MuJoCo and probed with rays against what a bean can actually hit. MuJoCo collides a mesh as its convex hull, so a cone made of one mesh is solid, and the probe counts it that way. The checker never runs the simulation, so whether the gate holds the beans and then lets them all out is judged by Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, each shown the brief, the figure, the printed count and the rule, and never which model made it. On 2026-10-16 all three agreed on all ten results, and with the verdicts first published, for $0.52. The rule was written after Claude Opus 5.5 had looked at the runs. Every model on this task.
| check · how it is decided | run 2 | run 3 |
|---|---|---|
| Serves all sixmeasured · The tray hands each of the six tables a drink it will take, every drink used exactly once and none twice. A refusal leaves it undecided, and a tray that reuses a drink fails it. | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Six drinks with fixed attributes, six tables with fixed notes, one tray. The checker derives who can take what, reads the tray's labels, and confirms with a perfect-matching solver that a serving exists: A to F in order. A refusal on a solvable brief is wrong, and so is a tray that reuses a drink or hands a table a drink it will not take. Whether a drawn cup really holds what its label says is the eye's. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Says it cannot be donemeasured · The reply, or a comment in the file, says no serving of all six exists. A run that draws a tray instead and says nothing fails it. | ✓ pass | ✓ pass | ✓ pass |
| Reuses nothingmeasured · If it drew: no drink given to two tables and no table given two drinks. A run that drew nothing is undecided. | · n/d | · n/d | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Tables 1, 2, 3 and 4 together will only take the three drinks A, B and C, so no serving of all six exists, and the checker confirms that with its own perfect-matching solver. The right answer says so and draws nothing. Every table still has a drink it will take on its own, so the shortage is only visible across the tables together. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Serves all sixmeasured · The tray hands each of the six tables a drink it will take, every drink used exactly once and none twice. A refusal leaves it undecided, and a tray that reuses a drink fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v2, checks c1, harness h1. Ground truth: Two tables want black coffee and table 4 has to take the iced oat latte or tables 1, 2 and 3 run out of A, B and C. A serving exists, A to F in order, and the checker confirms it with its own matching solver. Refusing because it looks crowded is wrong. Every model on this task.
| check · how it is decided | run 1 | run 2 | run 3 |
|---|---|---|---|
| Same filemeasured · 48 kHz, the same length within 10 ms, and no sample at full scale. | ✓ pass | ✓ pass | ✓ pass |
| Hum downmeasured · The 50 Hz lines from 40 Hz to 16 kHz, within 0.5 Hz of each, at least 10 dB quieter than the pump on its own puts there. | ✗ fail | ✗ fail | ✗ fail |
| Rush downmeasured · From 0.3 to 4.5 s, everything off the lines at least 3 dB quieter than the pump on its own puts there. | ✗ fail | ✗ fail | ✗ fail |
| Music keptmeasured · Where the music is strong and the pump weak, each of four bands from 40 Hz to 8 kHz within 1.5 dB of the music, with no level correction. | ✗ fail | ✗ fail | ✗ fail |
| Sounds rightby eye · Pass if, played straight after the music on its own, no note is missing or dulled and nothing new can be heard: no warble, no clicks, no hollow sound. Some hum or rush left over does not fail it, since Hum down and Rush down score that. | ✗ fail | ✗ fail | ✗ fail |
Prompt v1, checks c1, harness h1. Ground truth: cafe.wav is 18.2 s of the Espresso Rag with a CC0 recording of a Nespresso pump 6 dB under it, and the checker holds the music and the pump apart. The pump is a vibration pump on 50 Hz mains: lines at every multiple of 50 Hz up past 14 kHz, which carry 89% of its energy, and a rush of water that is loudest in the first four seconds. Every model on this task.