vc_018 · Cross-surface — chat → ML → chat

Status: ✓ Passing · Last run: 2026-06-04T23:50:46 · Pipeline: cross_surface · History: 2/3 runs passed (67%)

What was tested

End-to-end test of the most common user journey: user chats with the agent asking it to build a model, model trains in the background, user returns to chat asking what the model found. Three things have to all work: 1. Chat correctly invokes start_ml_run on turn 1 (right tool, not a fabricated answer) 2. v2 trains a model and surfaces the planted driver 3. Chat turn 2 reads the model's interpretation HTML from the session and accurately reports which feature matters

Failure modes this covers: - Agent answers turn 1 from "knowledge" without queuing a build - v2 runs but never writes results back to the session - Chat turn 2 hallucinates feature names not in the trained model

Uses entity_static (same shape as vc_001) so v2's behavior on this data is known-clean — any failure must be in the surface handoff.

What we planted in the data

• 2,000 entities
• Planted driver: usage_score
• 3 noise feature(s) (no relationship to outcome)
• Target positive rate ≈ 30%
• Signal strength: 0.70 (sigmoid slope multiplier)

What the system did

• Features used in model: 8
• Model selected: LogisticRegression
• CV AUC: 0.913

Top features by importance:

• usage_score_raw (1.653)
• noise_2_raw (0.068)
• noise_3_raw (0.035)
• noise_1_raw (0.019)
• noise_2_present_flag (0.000)

What we checked — all assertions passed

How this could have gone wrong

• If tool_was_used failed: Agent must call start_ml_run on the first turn — not invent an answer about training. Scope=turn (not session) because this assertion specifically checks the handoff happened on turn 1.
• If planted_feature_in_top_k_importance failed: v2 trained a model and recovered the planted signal. This is the same assertion vc_001 makes — what differs is that here v2 was invoked from chat, not from the test directly.
• If auc_in_band failed: AUC consistent with signal_strength=0.7. Sanity-checks the v2 run produced a real model, not a degenerate one.
• If tool_was_used failed: On turn 2, the agent shouldn't need any tool call — the model's interpretation_html was injected into the session prompt. But using profile_data or query_bigquery is acceptable. We pass a list so either approach (read-from-prompt OR re-investigate- the-data) is OK.
• If answer_mentions_fact failed: The agent's reply must surface usage_score as the top driver — that's what v2 trained. If the agent says something else (e.g. a noise_* feature), either v2 trained badly OR chat hallucinated.

Why this case matters

Every other case tests ONE surface. This is the only case that proves the two surfaces fit together — the handoff to v2 happens cleanly, v2 writes back to where chat reads from, chat can intelligibly answer questions about the model. That handoff is where production bugs hide; it's also the customer's actual journey.

Reproducing

# from auto_insight_api/
python -m validation.v2 run vc_018 --pipeline cross_surface -v

• Case config: validation/v2/cases/18_cross_surface.yaml
• Data shape: entity_static
• Analytics type: cross_surface

← back to validation report