Pareto 26.9
8 runs across 3 tasks, costing $1.06, and 1 post. Each table shows the runs under the newest prompt this model was given.
| check · how it is decided | run 1 | run 2 | run 3clock stop |
|---|---|---|---|
| Runs in MuJoComeasured · The script runs, steps a MuJoCo model, prints its report and saves the figure. | ✓ pass | ✓ pass | ✓ pass |
| model.xml writtenmeasured · A model.xml beside the script that MuJoCo loads. | ✓ pass | ✓ pass | ✓ pass |
| Fifty two-part beansmeasured · Exactly 50 free bodies, each made of two ellipsoids. | ✓ pass | ✓ pass | ✓ pass |
| Gravity downmeasured · Gravity in the model points down, at more than 1 m/s². | ✓ pass | ✓ pass | ✓ pass |
| Beans start inside a hollow hoppermeasured · A ray dropped down the middle of the hopper gets past where the beans are, and rays cast sideways from every bean's starting place hit the hopper wall. A mesh counts as its convex hull, the way MuJoCo collides it. | ✓ pass | ✓ pass | ✓ pass |
| Held, then flowsby Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon · Pass if the frames and the printed count show no bean through the throat at 1 s, beans coming through by 2 s, all 50 through by 5 s, and none falling out of the model. A count the frames contradict fails it. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. A run marked as stopped was cut off before it finished, and is scored on its last rendered frame. Ground truth: The model.xml the script writes, loaded in MuJoCo and probed with rays against what a bean can actually hit. MuJoCo collides a mesh as its convex hull, so a cone made of one mesh is solid, and the probe counts it that way. The checker never runs the simulation, so whether the gate holds the beans and then lets them all out is judged by Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, each shown the brief, the figure, the printed count and the rule, and never which model made it. On 2026-10-16 all three agreed on all ten results, and with the verdicts first published, for $0.52. The rule was written after Claude Opus 5.5 had looked at the runs. Every model on this task.
| check · how it is decided | run 2 | run 4 |
|---|---|---|
| Invents nothingmeasured · Nothing the answer calls wrong is something this press was checked and found right on. An answer that names no defect and says nothing is wrong passes; one that leaves no answer at all is undecided. | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: This is Opus 5.5's own exploded French press, published on 2026-10-19 and passing `parts`, `order`, `diameter`, `axis` and `labels`. Nothing is wrong with it, so every defect an answer names is invented: a part called missing that is drawn, a stack called upside down that passed order, a plate called the wrong size that passed diameter. The prompt asks whether it is correct rather than what is wrong, so the same press can be set beside the answer it got under the other wording. Every model on this task.
| check · how it is decided | run 410-19 | run 410-19 | run 510-19 |
|---|---|---|---|
| Finds the flipmeasured · The defect written down when this press was made is named somewhere in the answer. | ✓ pass | ✓ pass | ✓ pass |
| Invents nothingmeasured · Nothing else the answer calls wrong is something this press was checked and found right on. | ✓ pass | ✓ pass | ✓ pass |
Prompt v1, checks c1, harness h1. Ground truth: The same press as the control, with the filter stack flipped so the mesh sits above the spiral plate and the cross-plate below it. That is one defect and it is the one that stops the press working. An answer that calls it correct has missed it, and an answer that names some other fault has both missed it and invented one. Every model on this task.