cc_002 · Canonical: v2 pipeline AUC at strong signal (locked exact)

Status: ✓ Passing · Last run: 2026-06-11T05:10:50 · Pipeline: v2 · History: 1/2 runs passed (50%)

What was tested

Same setup as cc_001 but with signal_strength=0.9 — a near-deterministic binary outcome. The v2 pipeline should recover a high AUC; the answer key locks the exact value. Paired with cc_001 (mid signal=0.5) and cc_003 (weak signal=0.2), these three cases bracket the operational signal-strength range and lock AUC at three fixed points. If the pipeline ever stops differentiating by signal strength, all three cases drift together and the regression is unmistakable.

What we planted in the data

• 1,000 entities
• Planted driver: usage_score
• 3 noise feature(s) (no relationship to outcome)
• Target positive rate ≈ 30%
• Signal strength: 0.90 (sigmoid slope multiplier)

What the system did

• Training rows built: 1,000
• Features used in model: 10
• Model selected: LogisticRegression
• CV AUC: 0.956

Top features by importance:

• usage_score_raw (1.138)
• usage_score_sq (1.138)
• noise_1_raw (0.045)
• usage_score_below_neg1std_flag (0.000)
• noise_2_high_flag (0.000)

What we checked — all assertions passed

How this could have gone wrong

• If canonical_metric_match failed: Cross-validated AUC matches the locked value to ±0.02. Should be meaningfully higher than cc_001's locked value (the same dataset minus signal recovery is a much easier learning problem).

Why this case matters

cc_001 alone locks the mid-signal AUC. cc_002 (high) and cc_003 (low) add a third dimension: the pipeline's AUC has to vary monotonically with signal strength. If three runs all produce the same locked AUC regardless of signal_strength, the pipeline is broken — but each case alone wouldn't catch that.

Reproducing

# from auto_insight_api/
python -m validation.v2 run cc_002 --pipeline v2 -v

• Case config: validation/v2/cases/cc_002_canonical_auc_high_signal.yaml
• Data shape: entity_static
• Analytics type: ml_binary

← back to validation report