cc_003 · Canonical: v2 pipeline AUC at weak signal (locked exact)
Status: ✓ Passing · Last run: 2026-06-11T05:11:59 · Pipeline: v2
· History: 1/2 runs passed (50%)
What was tested
Same setup as cc_001 but with signal_strength=0.2 — close to random. The v2 pipeline must still recover a modest-but-above-random AUC; the answer key locks the exact value. Weak-signal AUC is the most sensitive: small changes in feature engineering or model regularization can wash out the signal entirely. Locking the weak-signal AUC is the strictest reproducibility test in the canonical suite.
What we planted in the data
- 1,000 entities
- Planted driver:
usage_score - 3 noise feature(s) (no relationship to outcome)
- Target positive rate ≈ 30%
- Signal strength: 0.20 (sigmoid slope multiplier)
What the system did
- Training rows built: 1,000
- Features used in model: 4
- Model selected:
LogisticRegression - CV AUC: 0.712
Top features by importance:
usage_score_raw(0.610)noise_1_raw(0.074)noise_3_raw(0.049)noise_2_raw(0.030)
What we checked — all assertions passed
| Status | Assertion | Detail |
|---|---|---|
| ✓ | canonical_metric_match |
cv_auc_mean=0.7116477272727274 matches answer key 0.697849025974026 (diff=0.013799, tolerance=0.02) |
How this could have gone wrong
- If
canonical_metric_matchfailed: Cross-validated AUC matches the locked value to ±0.02. Should be lower than cc_001's locked value.
Why this case matters
When a change to the pipeline silently regularizes more aggressively (e.g. someone increases L2 penalty), weak-signal AUC collapses to ~0.5 first. cc_003 catches this before strong-signal AUC even moves. The earliest tripwire.
Reproducing
# from auto_insight_api/
python -m validation.v2 run cc_003 --pipeline v2 -v
- Case config:
validation/v2/cases/cc_003_canonical_auc_low_signal.yaml - Data shape:
entity_static - Analytics type:
ml_binary