vc_020 · Event-stream entity table (Forj retention shape)
Status: ✗ Failing · Last run: 2026-08-24T17:41:40 · Pipeline: v2
· History: 1/2 runs passed (50%)
What was tested
Mirror of run 142a7e4d6ab549e3b1a14281adc1eb67 (2026-05-26), surfaced by the diagnostic replay sweep. The customer's table had many rows per client (monthly snapshots), with categorical columns like health_tier that the LLM tried to partition by. The fans_out detector correctly flagged the entity table as event-stream and pred_grid_plus skipped the JOIN — so health_tier never made it onto pg. Pre-fix: 16 agg_by health_tier features all died at materialize with "Name health_tier not found inside pg"; model trained on 1 surviving feature → AUC=0.507 (random); Firestore doc had empty error string (the original "no_error_string" mystery). Post-fix (feature_plan): _entity_side_columns now takes event_stream_tables and returns ONLY entity_id columns when the entity table is in that set. The LLM gets feedback rejecting the agg_by-health_tier features and iterates to a different shape.
What we planted in the data
- 200 entities
- 4 snapshots per entity
- Signal strength: 0.70 (sigmoid slope multiplier)
What the system did
- Training rows built: 800
What happened — assertion failures
| Status | Assertion | Detail |
|---|---|---|
| ✗ | no_pipeline_errors |
1 error(s): ["AuthenticationError: Error code: 401 - {'type': 'error', 'error': {'type': 'authentication_error', 'message': 'API key is i… |
| ✗ | auc_in_band |
no AUC in pipeline result |
How this could have gone wrong
- If
no_pipeline_errorsfailed: Pipeline must run cleanly. Pre-fix this LOOKED clean (no exception thrown) but produced a useless model — caught by auc_in_band below. - If
auc_in_bandfailed: Some signal must be recoverable. With propensity driving both n_active_users and churn, a working pipeline should land meaningfully above random. The pre-fix run got 0.507; the post-fix run should get well above 0.60 because the LLM iterates to features that actually survive dry-run.
Root cause and resolution
Errors raised:
AuthenticationError: Error code: 401 - {'type': 'error', 'error': {'type': 'authentication_error', 'message': 'API key is invalid.'}, 'request_id': None}
Resolution: this case is being actively debugged; see the validation platform's disclosure log for status.
Why this case matters
Customer retention / SaaS health-tier tables are an extremely common panel shape — multi-row per customer with categorical attributes. Before this fix, those datasets would silently produce random models with no clear error. Now: the validator catches the bad feature plan, the LLM iterates, and the model trains on features that actually exist on pg.
Reproducing
# from auto_insight_api/
python -m validation.v2 run vc_020 --pipeline v2 -v
- Case config:
validation/v2/cases/20_event_stream_entity_table.yaml - Data shape:
event_stream_entity_table - Analytics type:
ml_binary - Related: v2 run 142a7e4d6ab549e3b1a14281adc1eb67 (2026-05-26)