ic_002 · Industry: SaaS — customer churn (90-day)
Status: ✓ Passing · Last run: 2026-06-07T06:02:41 · Pipeline: v2
· History: 1/1 runs passed (100%)
What was tested
SaaS customer-retention shape: predict whether each customer will churn within the next 90 days. Five tables: customers + usage_events + subscriptions + billing + labels panel. Planted signal: declining usage + payment failures drive churn.
What we planted in the data
- 300 entities
- Signal strength: 0.70 (sigmoid slope multiplier)
What the system did
- Training rows built: 1,200
- Features used in model: 24
- Model selected:
RandomForestClassifier - CV AUC: 0.644
Top features by importance:
usage_events_recency_count_365d(0.187)export_recency_count_90d(0.164)usage_events_recency_count_180d(0.106)company_size(0.098)usage_events_recency_count_60d(0.087)
What we checked — all assertions passed
| Status | Assertion | Detail |
|---|---|---|
| ✓ | no_pipeline_errors |
clean run |
| ✓ | feature_survival_rate_above |
24/27 features survived (89% ≥ 40%) |
| ✓ | source_tables_contributing_above |
9 distinct source-table prefixes in surviving features (['billing', 'company', 'export', 'feature', 'payment', 'plan', 'signup', 'subscri… |
| ✓ | auc_in_band |
AUC = 0.644 inside band [0.55, 0.95] |
How this could have gone wrong
- If
no_pipeline_errorsfailed: Multi-table SaaS pipeline runs clean.
Why this case matters
Churn prediction is the most common SaaS use case. The data shape (customers + behavior events + subscription history + billing) is also the shape every CS-team-buyer's data lives in.
Reproducing
# from auto_insight_api/
python -m validation.v2 run ic_002 --pipeline v2 -v
- Case config:
validation/v2/cases/ic_002_saas_churn.yaml - Data shape:
industry_saas_churn - Analytics type:
ml_binary