Experiment report · Industry benchmark

A process that finds defects automatically whenever a trace is produced, and updates the next version of the benchmark

We read the traces of many models solving the same task to find task defects, fix the task, regrade and publish a new version of the benchmark. We counted where each verdict-changing defect was first found.

Verdict-changing defects	First found in traces	First found by automated gates
19	19 (15 conservatively)	0

Table 1. PLC and synthesized tasks.

Defects in the plant, runtime or grader change verdicts

Most verdict-changing defects sat in the plant, the runtime or the grader, and surfaced while reading failed runs. No defect found by reading the prompt alone is recorded as changing a verdict.

Limits

For 61% of defects we cannot yet tell whether they change verdicts (not rerun yet).
Only 19 cases.