Skip to content

Modelling parity ledger

The scorecard of the modelling parity hill climb (#9296): the capabilities an architect treats as table stakes in Revit, Archicad, Vectorworks and Bonsai, each measured by three probes. Rows, oracles and owners live in tests/modelling-parity/catalogue.json; statuses come from the probes' result files, scored by tests/modelling-parity/score.mjs.

Table-stakes score: 23 / 25 (met 21, partial 4, missing 0). A met row counts 1, a partial row 0.5. The score may only go up: node tests/modelling-parity/score.mjs <dirs> --check tests/modelling-parity/baseline/r2-combined.json fails when it or any row drops.

Per-level passes: SDK 20/25 · UI 21/25 · OUTPUT 23/25. Non-applicable levels earn no pass.

Summary: tests/modelling-parity/baseline/r2-combined.json, generated 2026-11-13T03:21:42.545Z from 6818 results on a7a16a70b2; inputs tests/modelling-parity/.out/r2-sdk, tests/modelling-parity/.out/r2-exercises, tests/modelling-parity/.out/r2-ui-probes, tests/modelling-parity/.out/r2-ui-exercises.

Levels:

  • SDK: a headless script on a blank model through bim.store / @ifc-lite/create, checked against a geometry or data oracle read back from the exported, meshed IFC (tests/modelling-parity/probes/sdk/).
  • UI: the same capability in the Model workspace on the real GPU via ifc-gpu-check (tests/modelling-parity/ui/).
  • Output: the SDK probe's IFC4 export, and any IFC a UI result hands over, through ifc-lite validate, a reopen round trip and IfcOpenShell validate with express rules (tests/modelling-parity/output/). Drawing files a UI result hands over (SVG, PDF, DXF, for the documentation rows) get non-empty and linework/page checks. SVG is tag-balanced; PDF and DXF structure is sanity-checked. Sheet DXF additionally gets independent ezdxf checks for closed rectangular wall boundaries, named layers, dimension linework and text, metre units and the known 104 mm span at 1:50. Other drawing checks remain sanity checks, not geometric reference comparisons.
  • Exercise: the best result of the F1–F6 exercises that exercise the row (reported, not scored).

✅ pass · ◐ partial · ❌ fail or error · – skipped, not run, or not applicable. A row is met when every applicable level passes, partial when any level passes or partly passes, missing otherwise; skip never counts as a pass.

Table stakes

Capability Group SDK UI Output Exercise Row Why not met
Storeys and levels (storeys) Levels and grids ✅ ✅ ✅ ✅ ✅ met
Structural grids (grids) Levels and grids ✅ ✅ ✅ ✅ ✅ met
Wall chains that auto-join (wall-chains-autojoin) Walls ✅ ✅ ✅ ✅ ✅ met
Layered (compound) walls (layered-walls) Walls ✅ ✅ ✅ ◐ ✅ met
Wall top and base constraints (wall-top-base-constraints) Walls ✅ ✅ ✅ ✅ ✅ met
Slabs from outlines, with openings (slab-with-openings) Slabs and roofs ✅ ✅ ✅ ✅ ✅ met
Roof by footprint (hip, gable, mono-pitch) (roof-by-footprint) Slabs and roofs ✅ ✅ ✅ ✅ ✅ met
Columns and beams on grids that follow levels (columns-beams-on-grid-follow-levels) Structure ✅ ✅ ✅ ✅ ✅ met
Curtain walls with grids, mullions and panels (curtain-walls) Facade ✅ ✅ ✅ ✅ ✅ met
Hosted doors and windows from types (hosted-doors-windows-from-types) Openings and circulation ✅ ✅ ✅ ✅ ✅ met
Stairs and railings between levels (stairs-railings) Openings and circulation ✅ ✅ ✅ ✅ ✅ met
Type editing that updates every instance (type-editing) Types and materials ✅ ✅ ✅ – ✅ met
Materials with appearance (materials) Types and materials ✅ ✅ ✅ ◐ ✅ met
Move, copy, rotate, mirror, array (transforms-move-copy-rotate-mirror-array) Editing ✅ ✅ ✅ ✅ ✅ met
Align, offset, trim/extend, split (align-offset-trim-split) Editing ✅ ✅ ✅ – ✅ met
Copy to levels (copy-to-levels) Editing ✅ ✅ ✅ ✅ ✅ met
Snapping and inference (snapping-inference) Precision input – ✅ – ✅ ✅ met
Typed input and temporary dimensions (typed-input-temp-dims) Precision input – ✅ – ✅ ✅ met
Rooms with computed area (rooms-with-area) Spaces and data ✅ ✅ ✅ ✅ ✅ met
Schedules and quantities (schedules-quantities) Spaces and data ✅ ◐ ✅ ✅ ◐ partial ui partial: group and footer totals refresh to 11 m after editing: null; task "edit one wall length to 6 m and read live totals" failed: Wall Length is read-only in Model inspector; cannot edit it to 6 m.
Plans, sections and elevations (plans-sections-elevations) Documentation – ✅ ✅ – ✅ met
Dimensions and tags (dimensions-tags) Documentation – ◐ ✅ – ◐ partial ui partial: a tag tool for elements (door, window, room tags): tag control found, but no entity-bound tag placement/export oracle has passed
Sheets, PDF and DXF export (sheets-pdf-dwg) Documentation – ◐ ✅ – ◐ partial Historical sheet evidence / oracle version unrecorded: the selected UI/Output evidence does not record the current PDF/DXF oracle version. Recorded statuses are retained; this baseline does not certify the current sheet target.
ui partial: drawing exported as DWG: no DWG export exists
Psets, classification and valid IFC4 (psets-classification-valid-ifc4) Spaces and data ✅ ✅ ✅ ✅ ✅ met
Large and tall model performance (large-tall-model-performance) Scale ✅ ◐ ✅ ✅ ◐ partial ui partial: single-edit median below 100 ms including remesh: {"count":5,"medianMs":145,"p95Ms":360,"maxMs":360}; exclusive benchmark admission: DIAGNOSTIC: PC benchmark admission is disabled; no timing qualification under the shared GPU helper

Beyond table stakes (reported, not scored)

Capability Group SDK UI Output Row
Curved walls (curved-walls) Beyond table stakes – – – ❌ missing
Ramps and sloped slabs (ramps) Beyond table stakes – – – ❌ missing
Ceilings (ceilings) Beyond table stakes – – – ❌ missing
Footings and foundations (footings) Beyond table stakes – – – ❌ missing
Groups (groups) Beyond table stakes – – – ❌ missing
Georeferencing (georeferencing) Beyond table stakes – – – ❌ missing
Terrain (terrain) Out of scope for this climb – – – ❌ missing
MEP systems (mep) Out of scope for this climb – – – ❌ missing
Multi-user co-editing (co-editing) Out of scope for this climb – – – ❌ missing
DWG export (dwg-export) Beyond table stakes – – – – not scored

Regenerating

node tests/modelling-parity/probes/sdk/run.mjs     # SDK probes -> tests/modelling-parity/.out/L1/sdk
node tests/modelling-parity/output/run.mjs         # output probes -> tests/modelling-parity/.out/L1/output
node tests/modelling-parity/score.mjs              # score every lane under tests/modelling-parity/.out
node tests/modelling-parity/score.mjs <dirs> --out tests/modelling-parity/baseline/<round>.json
node scripts/generate-modelling-parity-ledger.mjs  # this file, from the newest baseline summary