The gap between a model's best and worst task measures how hard the tasks are, which is not jaggedness. I was wondering what metric we could use for jaggedness, here is one that shows how uneven the model is between tasks.
First, the share of models that fail an evaluation is difficulty. Now if you sort every check from the one most models pass to the one most fail, and a smooth model passes a block on the left and fails the rest. The jaggedness metric J counts how often a model breaks that order from easy to hard tasks.
Take every pair of checks where i is easier than j, with p the share of its runs that passed each:
The top is failing the easier check and passing the harder one. The bottom is every pair where it passed one check and failed the other, either way round. So 0 is a perfect staircase the model passes easy tasks and fails difficult ones and 0.5 is a model which fails disregarding how difficult the task is.
| model | Venn: seven regions, three names | Four tables, three drinks: says it cannot be done | Bean cosy: a real chart | AeroPress: one assembly axis | Espresso Rag: left hand crosses | Espresso Rag: right hand keeps going | Hopper: model.xml written | Hopper: fifty two-part beans | Hopper: gravity down | Verdict, flipped: finds the flip | Verdict, flipped: invents nothing | Portafilter: two assembly axes | Bean cosy: stitches wider than tall | French press: labels match | French press: one assembly axis | Stirring spoon: handle stays on the bowl | V60: parts fit | Hopper: beans start inside a hollow hopper | Knitting repair: repaired | Venn repair: repaired | Venn: every drink in its region | Espresso Rag: stems apart | Portafilter: stack order | AeroPress: stack order | Verdict, good press: invents nothing | Portafilter: parts fit | Stirring spoon: the spoon does the stirring | Grinder: parts fit | V60: stack order | Bean cosy: bean knits true | Hopper: held, then flows | Café table: spoon reads as a spoon | Portafilter: all five parts | Café table: croissant reads as a croissant | French press: filter fits the glass | Stirring spoon: reads as inside the cup | AeroPress: all five parts | Five circles: names a fix that works | Pump noise: music kept | French press: all seven parts | Pruned shrub: flat side from the rules | Grinder: all eight parts | Spoon repair: in the drink | Venn: no label on an edge | Pruned shrub: sensible size | Espresso Rag: crossing marked | AeroPress: parts fit | Pruned shrub: reads as a shrub | Grinder: stack order | Pump noise: hum down | Pump noise: rush down | French press: stack order | passed | J, 90% range | p |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pareto 26.9 | 100% | - | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| GPT-6.1 Sol | 88% | 0.41 0.36 to 0.49 | 0.56 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Claude Sonnet 5.5 | 80% | 0.45 0.31 to 0.58 | 0.64 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Opus 5.5 | 80% | 0.39 0.30 to 0.48 | 0.35 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| DeepSeek 4.1 Flash | 74% | 0.52 0.41 to 0.64 | 0.85 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Pareto 26.10 Preview | 70% | 0.46 0.35 to 0.58 | 0.62 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Kimi K3.1 Pro | 68% | 0.15 0.11 to 0.19 | <0.01 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| GPT-6 Luna | 61% | 0.38 0.28 to 0.48 | 0.12 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| GLM 5.3 FlashX | 60% | 0.35 0.29 to 0.42 | 0.10 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Claude Haiku 5.5 | 55% | 0.30 0.24 to 0.36 | 0.01 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Command A+ | 24% | 0.39 0.26 to 0.50 | 0.59 | ||||||||||||||||||||||||||||||||||||||||||||||||||||
| Perceptron Mk1.5 | 9% | - | |||||||||||||||||||||||||||||||||||||||||||||||||||||
| all models | 98 | 96 | 95 | 93 | 92 | 92 | 92 | 92 | 92 | 90 | 90 | 86 | 86 | 85 | 84 | 83 | 83 | 83 | 78 | 78 | 76 | 75 | 74 | 70 | 70 | 69 | 67 | 67 | 67 | 67 | 67 | 63 | 62 | 58 | 58 | 56 | 53 | 53 | 53 | 53 | 52 | 50 | 44 | 42 | 42 | 42 | 40 | 39 | 33 | 33 | 33 | 31 | average pass rate, % | ||
Darker is passed more often. Checks run from the one most models pass to the one most fail, which the bottom row shows as each check's pass rate averaged over every model. Blank is a task the model has not run, and passed is the model's own average over its checks. J is how often, when the model passed one check and failed another, the one it failed was the easier: 0 is a perfect staircase and 0.5 ignores difficulty. The range is the middle 90% of J when its runs are redrawn. p is how often shuffling which checks it passed lands J as far from 0.5, so a small p means the order is not luck. Two checks equally hard count for neither side. A dash is a model that passed or failed only one check, where J cannot tell. Hover a J for the pair that breaks the order most. It counts every run so far, so it changes after the date of the post it sits in.
The p column says whether a J could be luck: shuffle which checks a model passed and see how often J lands as far from 0.5. So far only GLM 5.3 FlashX is clearly off 0.5, and it is smooth, so the board cannot yet call any model jagged.
Some models are great at doing hard things but poor at doing things that most models find easy.
Perceptron Mk1.5 gets a dash: it passes only the easiest check, and one pass only says where that check sits on the ladder. So J needs a model to pass and fail at least two checks each, which is where the metric stops working for very weak models.
The script is scripts/staircase.py in the evidence repo.