Naar de inhoud

Grok 4.7 reports 44.4% on CADGenBench, the public list comes out higher

This is the English reading version of the Dutch news item Grok 4.7 meldt 44,4% op CADGenBench, de ranglijst topt op 86,9%.

xAI has published the model card of Grok 4.7, including a CAD figure: 44.4% on CADGenBench. In xAI’s table that puts Grok at the top, above the models it compares itself with.

The leaderboard says something else

On the public list of submissions at CADGenBench, Claude stands much higher. There build123d-mcp 0.3.85 with Claude Opus 5.5 reaches 86.9% on the generation part, and 82.98% across the whole benchmark. That is nearly double the figure xAI quotes for Grok 4.7.

The difference is mainly in the setup, not only in the model. The top of that leaderboard works with an MCP server as the tool: the model drives a program that makes the geometry. xAI measures Grok with its own harness, and says so itself in the model card. So the figures do not come from the same measuring setup.

If you want to compare anyway, look at the rows that were set up the same way. Grok 4.6 stands at 41.5% on the leaderboard (Mecado Baseline with build123d) and at 40.9% in xAI’s table. Those two are close together.

My verdict: a table from the maker itself is not a leaderboard. Anyone who wants to run their tools themselves would not look at Grok either: that model is closed and works only through xAI’s API or through platforms xAI allows. The head of the public leaderboard is precisely a combination of an open-source MCP server with a model you pay for per use.

The figures are in the model card of Grok 4.7 and on the CADGenBench Generation leaderboard. What the benchmark itself measures is at CADGenBench.

Laat een reactie achter

Je e-mailadres wordt niet gepubliceerd. Vereiste velden zijn gemarkeerd met *