# Three CAD benchmarks: what they measure, and what they do not

- Link: https://vibecad.nl/en/three-cad-benchmarks-what-they-measure-and-what-they-do-not/
- Published: 2026-10-23T07:31:08+02:00
- Author: Marijn van den Heuvel \+ AI

_This is the English reading version of the Dutch news item [Drie CAD-benchmarks: wat ze meten, en wat niet](https://vibecad.nl/2026/10/23/drie-cad-benchmarks/)._

There are now three benchmarks that measure whether an AI model really does CAD 
work: Parametric CAD Bench, CADGenBench and AutoCAD Bench. Each measures something
different, and that is why you cannot line their figures up side by side.

How that is measured differs per benchmark.

| Benchmark | What it measures | 
| Parametric CAD Bench (gNucleus) | 100 FreeCAD tasks: 30 creating from text, 30 creating and then editing, 40 creating from an engineering drawing. The code really runs and the geometry is compared with the task and with an example model. | 
| CADGenBench (Hugging Face) | Whether a model turns a design task into a valid and geometrically correct 3D part. Invalid output gets no score. There are two kinds of task: generating and editing. | 
| AutoCAD Bench (Markov Studios) | 50 drawing tasks in AutoCAD 2019, 21 in 2D and 29 in 3D. The agent sees only a screen and may click, type and drag: no automation, no MCP. |

## Parametric CAD Bench: 61% with an agent around it

At the top of the V3 leaderboard is Claude Opus 5.5 at maximum effort with Claude
Code as its tool, at 61.03%. The same Opus reaches 57.96% with mini-swe-agent, and
GPT-6 Astra comes in with Codex at 56.87%. What stands out inside those figures:
going from an engineering drawing to CAD works best by far at 84.66%, and changing
an existing model stalls at 39.69%. The whole run of 100 tasks cost 573.94 dollars
in tokens at the model provider, about 5.74 dollars per task, and a failed or unscored
attempt counts as zero. Those costs are reported for the full cohort and depend 
on the model and agent configuration: in this list they run from 196.74 to 1,056.04
dollars.

## CADGenBench: that 86.9% is a partial figure

In reports about CADGenBench, 86.9% comes up for build123d-mcp with Claude Opus 
5.5. In the public list of submissions that is the figure on the generation part,
not the total: across the whole benchmark that submission comes to 82.98%, with 
77.0% on editing. At the top of the total score stands a submission named deeprd-
v2 at 84.82%, marked as not checked; the build123d-mcp submission is checked.

The benchmark runs each submission itself, but the participant supplies their own
combination of model and tools. So the list is a signal that a combination works,
not a ranking of models.

## AutoCAD Bench: 46%, and that is the highest

This one is strictest in what it allows: fifty tasks in AutoCAD 2019 in which the
agent has only a mouse and a keyboard, no MCP and no automation. At the top stands
Sol at 46%, so 23 of the 50 tasks, and the split shows where it goes wrong: 14 of
21 in 2D against 9 of 29 in 3D. Terra reaches 14%, Fable 5 10%, Luna 8%, and Opus
4.8, Kimi K2.5 and Qwen3.7 do not come above the threshold of 75 points in any task.

The makers say themselves what their benchmark does not see. The evaluator looks
only at the saved drawing file, not at how it was made. In their own example Sol
keeps the outline and every dimension but replaces two radius-driven arcs with a
single spline; that still got through with 86.3 points. What also stands out is 
that 88.8% of all actions are typed commands, and that the models often send several
actions from one screenshot.

## Why you cannot compare the figures

In one benchmark an agent writes FreeCAD code, in another it types and clicks in
AutoCAD itself, and in the third only the geometry that comes out counts. A figure
therefore says as much about the task and the setup as about the model, and the 
tool around it weighs in.

**My verdict:** the three lists together show something no single figure says. Models
are strong where they may write code and weak where they have to work in the interface
themselves: FreeCAD code goes reasonably, driving AutoCAD by hand stalls at 46%,
and that is still with an evaluator that looks only at the final file. For anyone
who wants to vibecad, that means the connection matters more than the figure: a 
program an agent can drive from outside gives it a route that works.

The leaderboards are at [cadbench.ai](https://cadbench.ai/), on the [CADGenBench leaderboard of Hugging Face](https://huggingface.co/spaces/HuggingAI4Engineering/CADGenBench),
and on the page of [AutoCAD Bench by Markov Studios](https://www.markovstudios.com/research/autocad-bench).
What CADGenBench measures exactly is in [What is CADGenBench?](https://vibecad.nl/en/what-is-cadgenbench/)