Modelling parity ledger¶
The scorecard of the modelling parity hill climb (#9296): the capabilities an architect treats as table stakes in Revit, Archicad, Vectorworks and Bonsai, each measured by three probes. Rows, oracles and owners live in tests/modelling-parity/catalogue.json; statuses come from the probes' result files, scored by tests/modelling-parity/score.mjs.
Table-stakes score: 23 / 25 (met 21, partial 4, missing 0). A met row counts 1, a partial row 0.5. The score may only go up: node tests/modelling-parity/score.mjs <dirs> --check tests/modelling-parity/baseline/r2-combined.json fails when it or any row drops.
Per-level passes: SDK 20/25 · UI 21/25 · OUTPUT 23/25. Non-applicable levels earn no pass.
Summary: tests/modelling-parity/baseline/r2-combined.json, generated 2026-11-13T03:21:42.545Z from 6818 results on a7a16a70b2; inputs tests/modelling-parity/.out/r2-sdk, tests/modelling-parity/.out/r2-exercises, tests/modelling-parity/.out/r2-ui-probes, tests/modelling-parity/.out/r2-ui-exercises.
Levels:
- SDK: a headless script on a blank model through
bim.store/@ifc-lite/create, checked against a geometry or data oracle read back from the exported, meshed IFC (tests/modelling-parity/probes/sdk/). - UI: the same capability in the Model workspace on the real GPU via
ifc-gpu-check(tests/modelling-parity/ui/). - Output: the SDK probe's IFC4 export, and any IFC a UI result hands over, through
ifc-lite validate, a reopen round trip and IfcOpenShell validate with express rules (tests/modelling-parity/output/). Drawing files a UI result hands over (SVG, PDF, DXF, for the documentation rows) get non-empty and linework/page checks. SVG is tag-balanced; PDF and DXF structure is sanity-checked. Sheet DXF additionally gets independent ezdxf checks for closed rectangular wall boundaries, named layers, dimension linework and text, metre units and the known 104 mm span at 1:50. Other drawing checks remain sanity checks, not geometric reference comparisons. - Exercise: the best result of the F1–F6 exercises that exercise the row (reported, not scored).
✅ pass · ◐ partial · ❌ fail or error · – skipped, not run, or not applicable. A row is met when every applicable level passes, partial when any level passes or partly passes, missing otherwise; skip never counts as a pass.
Table stakes¶
| Capability | Group | SDK | UI | Output | Exercise | Row | Why not met |
|---|---|---|---|---|---|---|---|
Storeys and levels (storeys) |
Levels and grids | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Structural grids (grids) |
Levels and grids | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Wall chains that auto-join (wall-chains-autojoin) |
Walls | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Layered (compound) walls (layered-walls) |
Walls | ✅ | ✅ | ✅ | ◐ | ✅ met | |
Wall top and base constraints (wall-top-base-constraints) |
Walls | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Slabs from outlines, with openings (slab-with-openings) |
Slabs and roofs | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Roof by footprint (hip, gable, mono-pitch) (roof-by-footprint) |
Slabs and roofs | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Columns and beams on grids that follow levels (columns-beams-on-grid-follow-levels) |
Structure | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Curtain walls with grids, mullions and panels (curtain-walls) |
Facade | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Hosted doors and windows from types (hosted-doors-windows-from-types) |
Openings and circulation | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Stairs and railings between levels (stairs-railings) |
Openings and circulation | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Type editing that updates every instance (type-editing) |
Types and materials | ✅ | ✅ | ✅ | – | ✅ met | |
Materials with appearance (materials) |
Types and materials | ✅ | ✅ | ✅ | ◐ | ✅ met | |
Move, copy, rotate, mirror, array (transforms-move-copy-rotate-mirror-array) |
Editing | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Align, offset, trim/extend, split (align-offset-trim-split) |
Editing | ✅ | ✅ | ✅ | – | ✅ met | |
Copy to levels (copy-to-levels) |
Editing | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Snapping and inference (snapping-inference) |
Precision input | – | ✅ | – | ✅ | ✅ met | |
Typed input and temporary dimensions (typed-input-temp-dims) |
Precision input | – | ✅ | – | ✅ | ✅ met | |
Rooms with computed area (rooms-with-area) |
Spaces and data | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Schedules and quantities (schedules-quantities) |
Spaces and data | ✅ | ◐ | ✅ | ✅ | ◐ partial | ui partial: group and footer totals refresh to 11 m after editing: null; task "edit one wall length to 6 m and read live totals" failed: Wall Length is read-only in Model inspector; cannot edit it to 6 m. |
Plans, sections and elevations (plans-sections-elevations) |
Documentation | – | ✅ | ✅ | – | ✅ met | |
Dimensions and tags (dimensions-tags) |
Documentation | – | ◐ | ✅ | – | ◐ partial | ui partial: a tag tool for elements (door, window, room tags): tag control found, but no entity-bound tag placement/export oracle has passed |
Sheets, PDF and DXF export (sheets-pdf-dwg) |
Documentation | – | ◐ | ✅ | – | ◐ partial | Historical sheet evidence / oracle version unrecorded: the selected UI/Output evidence does not record the current PDF/DXF oracle version. Recorded statuses are retained; this baseline does not certify the current sheet target. ui partial: drawing exported as DWG: no DWG export exists |
Psets, classification and valid IFC4 (psets-classification-valid-ifc4) |
Spaces and data | ✅ | ✅ | ✅ | ✅ | ✅ met | |
Large and tall model performance (large-tall-model-performance) |
Scale | ✅ | ◐ | ✅ | ✅ | ◐ partial | ui partial: single-edit median below 100 ms including remesh: {"count":5,"medianMs":145,"p95Ms":360,"maxMs":360}; exclusive benchmark admission: DIAGNOSTIC: PC benchmark admission is disabled; no timing qualification under the shared GPU helper |
Beyond table stakes (reported, not scored)¶
| Capability | Group | SDK | UI | Output | Row |
|---|---|---|---|---|---|
Curved walls (curved-walls) |
Beyond table stakes | – | – | – | ❌ missing |
Ramps and sloped slabs (ramps) |
Beyond table stakes | – | – | – | ❌ missing |
Ceilings (ceilings) |
Beyond table stakes | – | – | – | ❌ missing |
Footings and foundations (footings) |
Beyond table stakes | – | – | – | ❌ missing |
Groups (groups) |
Beyond table stakes | – | – | – | ❌ missing |
Georeferencing (georeferencing) |
Beyond table stakes | – | – | – | ❌ missing |
Terrain (terrain) |
Out of scope for this climb | – | – | – | ❌ missing |
MEP systems (mep) |
Out of scope for this climb | – | – | – | ❌ missing |
Multi-user co-editing (co-editing) |
Out of scope for this climb | – | – | – | ❌ missing |
DWG export (dwg-export) |
Beyond table stakes | – | – | – | – not scored |
Regenerating¶
node tests/modelling-parity/probes/sdk/run.mjs # SDK probes -> tests/modelling-parity/.out/L1/sdk
node tests/modelling-parity/output/run.mjs # output probes -> tests/modelling-parity/.out/L1/output
node tests/modelling-parity/score.mjs # score every lane under tests/modelling-parity/.out
node tests/modelling-parity/score.mjs <dirs> --out tests/modelling-parity/baseline/<round>.json
node scripts/generate-modelling-parity-ledger.mjs # this file, from the newest baseline summary