vc_003 · Leak guard
Status: ✓ Passing · Last run: 2026-06-04T23:36:49 · Pipeline: v2
· History: 6/7 runs passed (86%)
What was tested
Entity-static dataset with a planted, innocuously-named leaker column (engagement_index) that is essentially the target with tiny cover noise. The case tests the system's response to label leakage: EITHER the leak is dropped (gate/sanity catches it), OR the leak survives and the sanity check surfaces a 'possible label leakage' warning. The unsafe path — a near-perfect model shipped silently — must be catchable.
What we planted in the data
- 5,000 entities
- Planted driver:
usage_score - 3 noise feature(s) (no relationship to outcome)
- Target positive rate ≈ 30%
- Signal strength: 0.70 (sigmoid slope multiplier)
leak_strength=0.05
What the system did
- Training rows built: 5,000
- Features used in model: 4
- Model selected:
LogisticRegression - CV AUC: 0.902
Top features by importance:
usage_score_raw(2.384)noise_3_raw(0.027)noise_2_raw(0.019)noise_1_raw(0.003)
What we checked — all assertions passed
| Status | Assertion | Detail |
|---|---|---|
| ✓ | high_auc_implies_leakage_warning |
AUC 0.902 < threshold 0.99 — implication holds vacuously (no high-AUC = no leakage warning required) |
| ✓ | leak_feature_dropped_or_flagged |
leak engagement_index not in importance ranking — dropped before training |
| ✓ | no_pipeline_errors |
clean run |
How this could have gone wrong
- If
high_auc_implies_leakage_warningfailed: Conditional / Δ-test: IF the pipeline produced AUC ≥ 0.99, a leakage warning must be in the issues list. Vacuously true when AUC < threshold (no high-AUC = nothing to flag). What this catches: a near-perfect model produced without any safety flag — i.e., the silent-bad-model failure mode. - If
leak_feature_dropped_or_flaggedfailed: The planted leak column (planted.leak_feature='engagement_index') must either be absent from the importance ranking (gate dropped it) or accompanied by a leakage warning. Fail mode: leak ranks top-k AND no warning surfaced. - If
no_pipeline_errorsfailed: The leak case should still produce a model + result. We're testing detection behavior, not refusal — the pipeline should not crash.
Why this case matters
Label leakage is the most dangerous bug class for ML products: the model looks excellent in offline metrics but produces worthless or misleading predictions in production. The system must either catch leaks itself (gate/sanity) or surface enough information that a human reviewer can. This case anchors that property as a regression check.
Reproducing
# from auto_insight_api/
python -m validation.v2 run vc_003 --pipeline v2 -v
- Case config:
validation/v2/cases/03_leak_guard.yaml - Data shape:
entity_static_with_leak - Analytics type:
ml_binary - Related: __ml_sanity_check (AUC ≥ 0.99 flag), _gate_features_by_signal (signal threshold + redundancy)