vc_016 · High-NULL demographic column

Status: ✓ Passing · Last run: 2026-06-04T23:45:15 · Pipeline: v2 · History: 2/2 runs passed (100%)

What was tested

Entity-static dataset where one demographic column (insurance_type) is ~50% NULL — typical for customer data where many users decline to provide certain fields. The clean planted driver (usage_score) is still present; the test is whether the pipeline handles the NULL-heavy column gracefully. Acceptable behaviors: 1. Drop insurance_type at signal-gate (signal-to-noise too low with 50% missing — usage_score still wins on signal recovery) 2. Keep it but encode NULL as its own category — model should still discover usage_score as the primary driver Rejected behaviors: - Crash on NULLs during feature materialization - Allow a NULL-flag feature to dominate importance (would mean 50%-of-rows-NULL accidentally correlates with target by chance, which a working signal-gate should filter)

What we planted in the data

• 2,000 entities
• Planted driver: usage_score
• 2 noise feature(s) (no relationship to outcome)
• Target positive rate ≈ 30%
• Signal strength: 0.70 (sigmoid slope multiplier)
• null_rate = 0.5

What the system did

• Training rows built: 2,000
• Features used in model: 10
• Model selected: RandomForestClassifier
• CV AUC: 0.925

Top features by importance:

• usage_score_raw (0.392)
• usage_score_zscore_by_insurance (0.389)
• usage_score_centered_by_insurance (0.095)
• usage_score_percentile_by_insurance (0.086)
• usage_score_ratio_to_insurance_avg (0.028)

What we checked — all assertions passed

How this could have gone wrong

• If row_count_eq_n_entities failed: df has exactly one row per entity. Catches join-induced row-inflation that NULL-heavy columns sometimes trigger.
• If planted_feature_in_top_k_importance failed: The clean planted driver must still rank in top-3, regardless of how the pipeline treats insurance_type. If insurance-derived features outrank usage_score, something's wrong — those have no planted signal.
• If auc_in_band failed: AUC consistent with signal_strength=0.7. Below 0.70 means the pipeline got confused by the NULL column and lost signal; above 0.95 means a leak.
• If no_pipeline_errors failed: Clean run — NULL values must not crash feature_materialize or training. The most common NULL-related bug is "passed NaN to a step that didn't handle it" producing a stacktrace.

Why this case matters

Demographics columns at high NULL rates are universal in customer data — phone_carrier, referral_source, secondary_email all routinely have 40-70% missing. The pipeline must degrade gracefully: ideally drop or impute, never crash, never let NULL-derived noise outrank real signal.

Reproducing

# from auto_insight_api/
python -m validation.v2 run vc_016 --pipeline v2 -v

• Case config: validation/v2/cases/16_high_null_column.yaml
• Data shape: entity_high_null_column
• Analytics type: ml_binary

← back to validation report