Experiment report · Industry benchmark

Claude models cost more

We set each model’s cost of one full benchmark run on the public leaderboard (ef1.2: 100 tasks × 3 runs) beside its Hard pass rate. Against GPT models with a similar pass rate, Claude models cost far more: Opus (60% Hard) costs 18 times GPT-6.1 Terra (58%), and Fable (56%) 121 times. GPT’s strongest, Astra, is expensive too.

Model	Hard pass rate	Full-bench cost (GPT-6.1 Sol = 1)
GPT-6 Astra	72%	4.2×
GPT-6.1 Sol	70%	1.0×
Opus 5.5	60%	3.3×
GPT-6.1 Terra	58%	0.18×
Fable 5.1	56%	21.8×
Sonnet 5.5	43%	1.4×

Table 1. The six models with the highest Hard pass rate. Cost is the list-price total as a multiple of GPT-6.1 Sol’s.

Measurement conditions move the ranking too

Some early Claude runs were graded inside Harbor with a time limit. Taking those gradings as they were would drop Opus’s Hard pass rate from 60.4% to 42.9% and its rank from 3rd to 5th. The public leaderboard already uses offline grading without a limit. The other models were only ever graded offline, so the same comparison cannot be made for them.

Limits

Costs are API list-price equivalents, not invoices.
Harnesses differ (Claude Code, Codex), and the cost gap includes that difference.