Failure types

Every failed check in Three Shots gets one of these, so the same mistake made by two models on two different tasks ends up in the same place. 76 of 144 results carry at least one.

So which number should you read? How many models a type has appeared in. One model failing one way tells you about that model, and several models failing the same way on one task tells you about the task. But several models failing the same way across different tasks is the closest thing here to a jagged-frontier finding, though a new release can disprove it overnight.

The map

Each cell is how many of that model's runs failed a check this way, out of its runs on the tasks where this failure has happened at all. A grey 0 means it was tested and never failed this way, and a dot means it never ran those tasks. A column filled all the way down is a failure that survived a change of model.

How a check is decided in the first place