Experiment report · Simulator
Give models a PLC simulator and their pass rate jumps
Three models write PLC programs for 10 Hard tasks, once without a simulator and once able to run their own code in one and revise it. Same day, same image, same harness. Grading checks every condition, hidden ones included.
| Model | Without | With simulator | Fisher p |
|---|---|---|---|
| GPT-6 Astra | 21/30 | 30/30 | 0.002 |
| GPT-6.1 Sol | 13/30 | 29/30 | <0.001 |
| Opus 5.5 | 12/30 | 21/30 | 0.037 |
| Total | 46/90 (51%) | 80/90 (89%) | +38 pts (95% CI +23 to +52) |
Feedback, not information
Hiding the plant’s numbers in the simulator barely moved it (80/90 to 77/90). The gain comes from running and revising, not from the numbers.
22 of 30 cells went up
Of 30 model-by-task cells, 22 went up and 2 went down (sign p<0.001).
Limits
- Three models and ten Hard tasks.
- Agents could run only the base condition.