Skip to content

Validating against a reference

English | Japanese

« Home

Validation in this gallery is not a mechanism for "claiming correctness" but a mechanism for "displaying what evidence exists". This page explains its vocabulary — tier, gate, measured, audit — with a worked example of 1 case climbing the tiers.

Tiers T0–T3 — a display of the strength of evidence

tier Meaning Evidence required
T0 Comparison with an analytical solution, with evidence of convergence L2 error + order of convergence (e.g. the L2 and order of poiseuille)
T1 Published reference, with an automatic ctest gate Gate name and tolerance
T2 Published reference, manual single-point comparison Measured value and rel. err, or "not measured" if absent
T3 Demo / qualitative only Stated as such

A tier is not a grade of "correctness". The same correct computation is T0 if the order of convergence is measured, and T2 if only a single point is — a tier displays only the strength of the evidence that was measured. The current breakdown is 9 at T0, 9 at T1, 50 at T2 and 3 at T3, available from the By tier axis of Index and from Validation status.

Gates direct / module — why module gates exist

A gate is a golden ctest test, and there are 2 kinds. direct means ctest runs the benchmark itself; currently there is only 1, cavity3d_d3q27_ghia. module means the case inherits the gate of the physics module it uses; 28 cases fall under this (grouped under gate names such as closures, thermal_ddf, two_phase_laplace and free_surface). The remaining 42 cases have no gate.

The reason module gates exist lies in the compositional architecture itself: cases are written as compositions of openrheo_amr/*.hpp modules, so if the physics is validated 1 time on the module side, every case that plugs it in inherits the validation. The 🧩 badge on a page shows this inheritance, and which module carries which gate is tabulated in Models.

measure-then-pin — how numbers get onto a page

A comparison value on a page is "printed only after it has been run and recorded". The procedure: run the case, and record the QoI in the measured block of cases.json as the 4-item set value / rel_err / ref / provenance (example: "provenance": "harvest N=128" for advection-diffusion2d). Page generation prints the value and rel. err from this record, and cases with no record are published still showing "not measured" — in the words of the design document, "publish with [not measured] rows visible — that IS the honesty story". A number with only a value written down and no provenance does not exist in this wiki.

The honesty rule is a specification, not a defect report

  • The Schäfer–Turek comparison of cylinder2d has St inside the reference band, Cd_max at +8.7% (Cl_max +10.7%); this excess is not hidden but published in the results section of the note, together with the reason (the effective-diameter error of the staircase mask). Writing down even "why some quantities agree and others do not" is the job of the results section.
  • For the Cd of forest-cylinder-st falling 7.55% outside the DFG band, it was decided at design time to "publish with ⚠ and a 1-line diagnosis. Do not hide it, do not silently rerun" (wiki-design.md).
  • Being non-empty is normal for Known issues. Failed runs and runs cut short become pages, with that fact stated explicitly.
  • Sections with no artefact are omitted. They are never faked (the generator's house style, wiki_pages.py).

A case climbing the tiers — porous-poiseuille2d as an example

porous_poiseuille2d (Brinkman porous channel) is a model of how to climb. As its source header says, the reference is not code but an exact solution — the cosh profile of the Brinkman equation u(y) = (GK/ν)[1 − cosh((y−H/2)/√K)/cosh(H/(2√K))] — and the case prints its own L2 error for each Darcy number.

  1. T3: a picture comes out. If all you have is "it looks porous", you stop here.
  2. T2: measure and record 1 single-point comparison against a reference (an analytical solution or a published value).
  3. T1: the Darcy drag in this case is a coefficient closure in closures.hpp, so it inherits the module-gate ctest closures (orheo-test-closures in native/CMakeLists.txt). This is exactly where the current registration stands — tier: T1, gate_kind: module, gate_name: closures (gallery_index.json).
  4. To climb to T0, having an analytical solution is not enough — you must refine the grid, measure the order of convergence, and attach it as evidence.

That it is "T1 despite having an analytical solution" is not a defect but a consequence of the rule that tiers count only measured evidence.

Drift detection — how you learn that a validation has gone stale

As designed in wiki_pages.py, a page is made of 3 layers: derived (tables and commands; regenerated on every build, so they cannot go stale), measured (figures and numbers; pinned to 1 run that records the commit it ran at, with the run_id and run date printed on the page), and authored-once (prose; pinned to the source_sha of the source read at the time of writing). When the source moves, wiki_pages.py audit lists the affected notes and the _page notes whose evidence files have newer commits. In other words, "the validation is stale" never happens silently; it always shows up in the audit output — every page that claims agreement with a reference rests on this mechanism.