Finding
In Taking Jaggedness Seriously, Helen Toner says that models get good first at tasks a computer can check, because that is what they can be trained on. Every check here is either measured by a script or decided by my eye. So we now have a script to count and see if it agrees with that thesis.
Across all 327 runs, measured checks passed 82% of the time and eye checks 72%, so that checks out.
Verdict
Note: On the eight tasks that have both kinds, it flips: measured checks passed 64% and eye checks 78%. But that is a small sample so need to have more diverse tasks.
n = . No test.
Which checks fell first, task by task.