vc_002 · Snapshot panel
Status: ✓ Passing · Last run: 2026-06-05T03:12:59 · Pipeline: v2
· History: 9/13 runs passed (69%)
What was tested
Monthly client panel — one row per (client, snapshot_date), 200 clients over 24 months = 4,800 rows. A precomputed binary target per row, driven by usage_score. This is the Forj-shape that exposed the join fan-out: the test asserts the pipeline adopts the source rows directly (grid-provenance) and the feature join stays 1:1 instead of inflating rows ~24× via entity-only join.
What we planted in the data
- 200 entities
- 24 snapshots per entity at 30-day cadence
- Planted driver:
usage_score - 3 noise feature(s) (no relationship to outcome)
- Target positive rate ≈ 30%
- Signal strength: 0.60 (sigmoid slope multiplier)
What the system did
- Training rows built: 4,800
- Features used in model: 18
- Model selected:
RandomForestClassifier - CV AUC: 0.857
Top features by importance:
usage_score_avg_client_all_time(0.452)usage_score_sum_client_all_time(0.331)usage_score_roc_60_180(0.025)usage_score_roc_30_180(0.024)usage_score_roc_30_60(0.023)
What we checked — all assertions passed
| Status | Assertion | Detail |
|---|---|---|
| ✓ | row_count_eq |
training rows = 4800, expected = 4800 |
| ✓ | planted_feature_in_top_k_importance |
derivative(s) of usage_score in top 3: ['usage_score_avg_client_all_time', 'usage_score_sum_client_all_time', 'usage_score_roc_60_180'] |
| ✓ | auc_in_band |
AUC = 0.857 inside band [0.65, 0.95] |
| ✓ | noise_features_below_planted |
best planted derivative usage_score_avg_client_all_time is rank 1; no noise above |
How this could have gone wrong
- If
row_count_eqfailed: Training df has exactly 4,800 rows (200 entities × 24 snapshots). A ~24× fan-out from an entity-only feature join would push this to ~115,000; the assertion catches that immediately. - If
planted_feature_in_top_k_importancefailed: The planted driver (or one of its named derivatives) must rank in the top 3 features by importance. - If
auc_in_bandfailed: AUC inside a band consistent with signal_strength=0.6. Below 0.65 means the signal wasn't picked up; above 0.95 means something is leaking (post-snapshot data into features). - If
noise_features_below_plantedfailed: No noise feature should outrank the best planted-derived feature.
Why this case matters
This is the case the panel-fan-out bug would have failed silently with ~115,000 rows and an inflated AUC. The validation gate catches it cleanly. It also exercises the grid-provenance probe + the snapshot- aware feature join + the chronological train/val split — three related safety mechanisms in one case.
Reproducing
# from auto_insight_api/
python -m validation.v2 run vc_002 --pipeline v2 -v
- Case config:
validation/v2/cases/02_panel_snapshot.yaml - Data shape:
panel_snapshot - Analytics type:
ml_binary - Related: panel fan-out fix (grid-provenance + snapshot-aware join)