Experiment report · CAD

CAD tasks GPT-6 Astra gets 0% on

Models get a 2D drawing and must build the 3D part (STEP). The drawing is first-angle, and depth is dimensioned from the part’s back face. The information is the same either way; only the face depth is measured from changes.

Figure 1. The same part seen from above. Depth d is measured from the back face.

What goes wrong

The projection and the words “back face” are read correctly. What flips is the direction of the depth axis: running it from the back toward the front makes the frame left-handed, and the part comes out mirrored.
A mirror image has the same volume, so a check by volume alone cannot catch it. Most GPT runs never laid their own model back over the drawing.
Opus renders its own model in the right projection and compares it with the drawing. The few times Opus failed, it was the same mirror image, checked against the drawing in the same wrong frame.

Numbers

Tasks	GPT-6 Astra	GPT-6.1 Sol	Opus 5.5
Same block, depth now from the back face	1/6	2/6	6/6
Six first-angle blocks, all depth from the back face	0/18	2/18	17/18
Six first-angle left-hand sheet-metal parts, datum = back face	0/18	4/18	19/24

Table 1. Passing runs / runs. In the six-block round, 34 of GPT’s 36 runs were mirror images. Of Opus’s 5 failures in the sheet-metal round, 3 were mirror images and 2 task defects. Mirror verdicts were verified by reflection comparison, with 0 grader errors (checked 3 November).

Limits

The tasks are single parts modelled on real drawing conventions, and there are few tasks and runs.
Conditions were not identical across models in every round; in some rounds Opus ran without image libraries.
This is one trap. On other kinds of CAD task all three models do well.