cc_001 · Canonical: v2 pipeline reproducibility on a fixed dataset

Status: ✓ Passing · Last run: 2026-06-11T00:51:57 · Pipeline: v2 · History: 1/4 runs passed (25%)

What was tested

Pins three metrics the v2 ML pipeline produces on a fixed entity-static dataset (n=1000, signal_strength=0.5, seed=42):

• cv_auc_mean — cross-validated AUC (locked ±0.01) - n_train_rows — rows that reached the training step (locked exactly) - n_features — features the pipeline used (locked exactly)

Each metric is recorded once into cc_001.json and compared on every subsequent run. If any one drifts — for ANY reason: training algorithm change, feature-engineering shift, library upgrade, random-seed handling, dictionary parser change — this case fires. Accepting the drift requires deliberately re-recording the answer key. This is the canonical layer's MVP: one case, three locked claims. Future canonical cases (different signal regimes, different data shapes) can layer in alongside.

What we planted in the data

• 1,000 entities
• Planted driver: usage_score
• 3 noise feature(s) (no relationship to outcome)
• Target positive rate ≈ 30%
• Signal strength: 0.50 (sigmoid slope multiplier)

What the system did

• Training rows built: 1,000
• Features used in model: 10
• Model selected: RandomForestClassifier
• CV AUC: 0.855

Top features by importance:

• usage_score_squared (0.498)
• usage_score_raw (0.475)
• noise_2_raw (0.027)
• noise_2_above_p75_flag (0.000)
• noise_3_above_median_flag (0.000)

What we checked — all assertions passed

How this could have gone wrong

• If canonical_metric_match failed: Cross-validated AUC matches the locked value to ±0.01. Catches every silent change in the ML compute path.
• If canonical_metric_match failed: Exactly 1000 rows reach training — no fan-out, no row loss in the ingest path.
• If canonical_metric_match failed: Exactly the recorded number of features survive dictionary processing and feature engineering. Catches changes in feature selection that would otherwise be masked by stable AUC.

Why this case matters

Property-based "AUC > 0.7" tells you the pipeline learned SOMETHING. Canonical "AUC == 0.812 ± 0.01" tells you the pipeline is producing the SAME computation it produced when you last validated it. The difference between "looks ok" and "actually reproducible." This case is the deterministic-output snapshot test for the v2 pipeline.

Reproducing

# from auto_insight_api/
python -m validation.v2 run cc_001 --pipeline v2 -v

• Case config: validation/v2/cases/cc_001_canonical_auc_mid_signal.yaml
• Data shape: entity_static
• Analytics type: ml_binary

← back to validation report