This is the English reading version of the Dutch explainer Wat is CADGenBench?.
In reports about AI and CAD, benchmark figures come up, and CADGenBench is one of them. It measures whether a model turns a design task into a real CAD model. Below is what is actually measured, and why you should take such a figure with a pinch of salt.
What it measures
CADGenBench asks a model to turn a task into a CAD model. Not a picture, and not a mesh: real geometry, a solid you can open in a CAD program and carry on editing. After that it checks whether that worked.
That makes it a different question from most AI tests. A model that can give a convincing description of a part does not get far here; it has to deliver something that works as a model.
How it judges
The judging is not the opinion of a chatbot but a test. First the model has to be a valid solid: closed, without loose faces or holes in the wall. Then the shape has to be right compared with what was asked for. And the build-up has to be right: is a hole in the right place, is the wall thickness right, do the faces meet.
That last part is exactly where models stumble. A model can look good from the outside and still fail, because the build-up underneath is wrong. For a CAD user that is recognisable: a part you cannot edit is worth little in practice.
Where it comes from
CADGenBench comes from Mecado and is on Hugging Face. It is a public benchmark with a leaderboard where results are submitted. There is also a variant that measures not only making a model but also editing an existing STEP file.
Why one figure says little
A benchmark can only be compared fairly if everyone measures under the same conditions, and that is not always the case here. xAI measured its Grok 4.7 with a tool of its own, while the models it compared itself with used theirs. Then the ranking says something, but not everything.
On top of that, the leaderboard is still thin. There are few models on it at the moment, and that makes a high position less meaningful than it looks. A score shows that a model can do something, not that it is the best.
Not the same as BenchCAD
BenchCAD is a different benchmark with a different setup. The figures of the two do not say the same thing, even though you often see them in the same list. So do not compare numbers from different benchmarks with each other.
The benchmark is at CADGenBench on Hugging Face, with the leaderboard at Mecado. The figures I mention here come from the providers’ own model cards.