vc_013 · High-cardinality categorical encoding

Status: ✓ Passing · Last run: 2026-06-04T23:41:54 · Pipeline: v2 · History: 2/2 runs passed (100%)

What was tested

Entity-static binary classification with a high-cardinality categorical column (branch_code, ~200 distinct levels for n_entities=2000). usage_score is the actual planted driver; branch_code is uncorrelated with target. Catches a real customer-pain failure mode: pipelines that naively one-hot encode high-card columns explode the feature space (200+ binary columns from a single source field), corrupting downstream training. The v2 pipeline should handle this via frequency/target encoding, top-K + 'other', or by dropping the column at the signal gate. All three are acceptable; one-hot explosion is not.

What we planted in the data

• 2,000 entities
• Planted driver: usage_score
• 2 noise feature(s) (no relationship to outcome)
• Target positive rate ≈ 30%
• Signal strength: 0.70 (sigmoid slope multiplier)
• n_levels = 200

What the system did

• Training rows built: 2,000
• Features used in model: 10
• Model selected: LogisticRegression
• CV AUC: 0.925

Top features by importance:

• usage_score_raw (2.544)
• usage_score_minmax_by_branch (0.240)
• usage_score_centered_by_branch (0.231)
• noise_1_raw (0.224)
• noise_2_minmax_by_branch (0.137)

What we checked — all assertions passed

How this could have gone wrong

• If row_count_eq_n_entities failed: df has exactly one row per entity. No fan-out from categorical encoding (a wide one-hot wouldn't change row count, but if the pipeline mistakenly long-pivoted, this would catch it).
• If feature_count_below failed: Final feature count must be ≤ 30. With 4 base features (usage_score + 2 noise + entity_id) plus a branch_code that naively one-hot expands to 200+ columns, anything past 30 indicates the high-card explosion failure mode. A pipeline that drops branch_code (signal gate), frequency-encodes it (1 col), or top-K + 'other' (≤ 11 cols) all sit well under 30.
• If planted_feature_in_top_k_importance failed: usage_score is the actual driver. Whatever the pipeline does with branch_code, the real signal must still surface as a top-3 feature.
• If auc_in_band failed: AUC consistent with signal_strength=0.7. If branch_code's noise corrupted training (e.g. via target-leakage in target-encoding without held-out folds), AUC could spuriously exceed 0.95.
• If no_pipeline_errors failed: A clean run — the pipeline survives the high-card column without crashing (memory blowup from explicit dense one-hot is a real failure mode on bigger datasets).

Why this case matters

High-cardinality categoricals are one of the top three customer-pain failure modes for ML pipelines (alongside leakage and panel fan-out). Without this case, "the pipeline handles real-world categorical data" is unverified — most synthetic test cases use low-cardinality segments (Enterprise/SMB/Mid) where naive one-hot is fine.

Reproducing

# from auto_insight_api/
python -m validation.v2 run vc_013 --pipeline v2 -v

• Case config: validation/v2/cases/13_high_card_categorical.yaml
• Data shape: high_card_categorical
• Analytics type: ml_binary

← back to validation report