Methodology

This document describes how the validation report is constructed: what each case is, what we assert, how we draw the line between "the system worked" and "the system silently broke," and how we disclose what we find.

The shape of a case

Every case is a synthetic dataset whose ground truth we know by construction, plus a set of structural and behavioral checks the platform must satisfy when it processes that dataset.

A case carries:

• A data shape — what the data looks like structurally (one row per entity? a clean panel? a two-table join?). This routes the platform's pipeline through different code paths, and the cases are chosen to exercise the paths whose past failures hurt the most.
• Planted truth — what we made true by construction. For example: a specific feature drives the target, three other features are noise, the positive rate is 30%.
• A set of assertions — properties the platform's output must satisfy if it interpreted the data correctly. These are deliberately property-based (relationships that must hold) rather than value-based (matching specific numbers), because the platform's LLM-authored steps produce equivalent-quality outputs in different specific shapes.

Cases live as YAML; the runner reads them, generates the synthetic data, runs the platform, and evaluates the assertions. No new code is needed to add a new case — only a new YAML and (if the shape is genuinely new) a small generator function.

The four kinds of assertions

Different things deserve different oracles. The validation framework uses four:

Invariant

Structural facts that must hold regardless of model quality — row counts, no fan-out, no duplication, schema validity, halt-on-degenerate-input. These are the cheapest checks and the first line of defense against silent breakage.

Example: row_count_eq_n_entities — for an entity-static dataset of N entities, the training table must have exactly N rows. If it inflates to 22×N, something joined wrong.

Signal recovery

Anchored to the planted ground truth. If we planted that feature X drives the target, the model must recover X in its top features. If we planted a positive rate of 30%, the realized rate after labeling must match.

Example: planted_feature_in_top_k_importance — the feature we made driving must appear in the top K importances. Derivations of the planted feature (the platform may rename or transform it) count as recovery.

Conditional (Δ-test)

Properties of the form "if X happens, Y must also happen." Useful for testing safety mechanisms: "if AUC ≥ 0.99, a leakage warning must surface." Vacuously true when the antecedent is false. Catches the unsafe path (high AUC produced silently without any flag).

Example: high_auc_implies_leakage_warning — used by the leak-guard case to verify that suspiciously-good results never ship without a sanity flag.

Canonical (reserved)

A small layer for absolute-truth claims that property-based assertions can't anchor — e.g., a specific KPI calculation that has one correct answer. Currently empty by design: the cases we have so far are well-served by the other three kinds. Reserved for cases where the claim is "this number is X," not "this property holds."

Why property-based, not value-based

A platform whose pipeline includes LLM-authored steps (intent classification, feature planning, label SQL) produces equivalent-quality outputs that differ in specific shape between runs. A test that requires byte-identical output is unstable for the wrong reason — it fails when the LLM picks 87 features instead of 99, even though both produce the same model quality.

Property-based assertions sidestep this: they describe what must be true about any correct output, not about this specific output. The framework's job is to encode "what makes an output correct" as checkable properties.

The trade-off, named honestly: properties can miss systematic biases that affect all outputs uniformly (if everything's off by 5%, no relationship-based property catches it). This is what the canonical layer exists for — when we need that bite, we hand-write the reference answer for a handful of cases.

How failures are disclosed

Once a case is shown externally, it has two paths: fix or disclose. We do not silently retire a failing case.

When a case fails:

1. The failure is captured in the result JSON (the raw record).
2. The hub page shows it in the "Recent failures" section.
3. The case's detail page narrates what happened, what the root cause was, and (once fixed) what release resolved it.

This means readers can see not only the green state but the history of what broke and how. A "100% pass" badge with no disclosed failures is a signal to distrust — we'd rather show our regressions transparently than imply we never have them.

What's in scope (and what isn't)

In scope today:

• Binary classification problems on the analytics platform (ml_binary).
• Structural shapes: entity-static, panel snapshots, multi-table joins, entity-static with planted leakage, degenerate-target halting.
• Properties: row counts, planted-signal recovery, leak-guard behavior, halt-on-degenerate.

Not yet in scope:

• Regression problems (ml_regression).
• Statistical models — hypothesis tests, confidence intervals, calibration over null populations.
• Dashboard outputs — aggregations, segmentations, time-window integrity.
• Cross-surface consistency (the same data flowing through both ML and dashboard surfaces).

These are planned. Until they're tested, we don't claim coverage on them.

How to interpret a pass rate

A case's pass rate over its run history is not an SLA. Runs accumulate as the framework itself develops, and historical failures from iteration during development stay in the record. The signal that matters is:

• The latest run's status (pass/fail).
• The trend across recent releases.
• Disclosed failures and their resolution timelines.

Don't read 75% historical pass rate as "the platform breaks 25% of the time" — it more often means "this case was iteratively developed and we kept its failed development runs in the record." If you want the production-quality signal, look at the latest-N runs, not the all-time rate.