Can AI agents run a factory?

# results

Capable, but not yet consistent

With three runs per task, up to 87% of the Hard tasks are passed at least once (GPT-6.1 Sol and Opus 5.5). Passing all three runs is rarer: at most 60% (GPT-6 Astra).

Best pass rate, 1 run

72%

GPT-6 Astra

Best within 3 runs

87%

GPT-6.1 Sol and Opus 5.5

Best in all 3 runs

60%

GPT-6 Astra

Never passed

3 of 30

Hard tasks, in 117 runs

Spending more buys little

GPT-6.1 Sol passes within 2.2 points of GPT-6 Astra and costs 4 times less per Hard run. The most expensive model, Fable 5.1 at $5.61 a Hard run, passes 16.7 points fewer than GPT-6 Astra.

The full leaderboard: safety and the index →

# tasks

100 tasks in two suites, each at three levels: Easy, Medium and Hard. A task is one factory work cell: its devices, an I/O map with exact addresses, an operating spec, a completion condition and safety rules. All come from PLCWorld; we audited and fixed them before ranking any model.

All 100 tasks, with replays →

# grading

A program grades every run: no LLM judge and no human grading. Each model works through its own harness (Codex, Claude Code, mini-swe-agent), three runs per task, and leaves one file. Here is one Hard task, MAT-H02, from the brief to the verdict.

1. 01

   The brief

   Each task describes one work cell: its devices, the I/O map with exact addresses, the operating spec, the completion condition and the safety rules.

   Brief

   MAT-H02

   Maintain accessibility of bypass storage in situations where the FIFO lead and the required types of two kits differ, and transfer actual parts as kit members of the current order.

   Inputs

   Outputs

   Conditions

   9

   Scan

   10 ms

   Signals

2. 02

   The program

   The agent leaves one IEC 61131-3 Structured Text file, /app/program.st. It gets no PLC compiler and no plant simulator, so the program has to be right before it ever runs.

   program.st

   GPT-6.1 Terra

   156CASE state OF157    0:158        IF NOT X11 AND D1 = 0 AND gen <= 2 THEN159            source := 0;160            dest := 0;161            IF D5 <> 0 AND X15 AND D13 < 2 AND D6 = D11 THEN162                source := 2; dest := 4;163            ELSIF D5 <> 0 AND X16 AND D14 < 2 AND D6 = D12 THEN164                source := 2; dest := 5;165            ELSIF D7 <> 0 AND X15 AND D13 < 2 AND D8 = D11 THEN166                source := 3; dest := 4;167            ELSIF D7 <> 0 AND X16 AND D14 < 2 AND D8 = D12 THEN168                source := 3; dest := 5;169            ELSIF X10 AND D3 <> 0 THEN

3. 03

   The plant

   A separate verifier compiles the file and runs it scan by scan, every 10 ms of plant time, in closed loop with a simulated plant.

   1. 1Sample the inputs
   2. 2Run the program
   3. 3Write the outputs
   4. 4Move the plant

   every 10 ms of plant time

   Repeated under 9 conditions for this task (476 across the benchmark), with device parameters, timings and disturbances varied inside the published ranges.

4. 04

   The verdict

   Task success and safety are judged separately in every condition. A run passes only if every condition reaches the goal with no safety violation; anything less scores 0.

   Condition	Goal	Safety
   1
   2
   3
   4
   5
   6
   7
   8
   9

   Run Pass

   Replay →

   Condition	Goal	Safety
   1
   2
   3
   4
   5
   6
   7
   8
   9

   Run Fail

   Replay →

   Two runs of MAT-H02: the replays on its task page.

# access

Run it, or add your model

The tasks, the verifier and the reference programs are public. Send us a model for the leaderboard, or a flaw you found in a task.