vc_010 · Statistical analysis — segment-mean significance
Status: ✓ Passing · Last run: 2026-06-04T05:56:25 · Pipeline: analyze
· History: 2/3 runs passed (67%)
What was tested
Same data shape vc_006 uses (entity_with_segments) but the question asks for a STATISTICAL test (t-test for difference of means), not a descriptive aggregation. The planted effect is large by construction — Enterprise mean revenue ≈ $120k vs SMB ≈ $5k, n≈100 vs n≈400, so any reasonable significance test returns p << 0.05. This case fills the statistical column of the coverage matrix without requiring a new pipeline backend: it routes through the same analyze_agent chat surface as vc_006-009, but the question shape forces the agent to use run_python (for the t-test or bootstrap), not just query_bigquery.
What we planted in the data
- 1,000 entities
- Signal strength: 0.50 (sigmoid slope multiplier)
What the system did
- Model selected:
analyze_agent
What we checked — all assertions passed
| Status | Assertion | Detail |
|---|---|---|
| ✓ | tool_was_used |
agent used run_python in session up through 'q_seg_diff_significant' |
| ✓ | answer_mentions_fact |
all expected facts present in answer to 'q_seg_diff_significant' |
How this could have gone wrong
- If
tool_was_usedfailed: Significance testing — t-test, bootstrap, or any inferential procedure — has to be computed in Python. If the agent answered "yes, significant" without ever running a test, it hallucinated. - If
answer_mentions_factfailed: The realized effect is gigantic (~20× difference in means with n_Enterprise ≈ 100, n_SMB ≈ 400) — any honest test returns p << 0.05. The agent's reply must say "significant".
Why this case matters
Statistical claims are where LLMs go wrong most quietly: the model knows the rhetoric of significance ("p-value", "confidence interval", "statistically meaningful") but can produce those words without actually computing anything. This case verifies that for an inferential question, the agent invokes run_python (the path where computation actually happens) — not just query_bigquery (where only data retrieval happens). Without this case, "the chat surface handles statistical questions" is unverified.
Reproducing
# from auto_insight_api/
python -m validation.v2 run vc_010 --pipeline v2 -v
- Case config:
validation/v2/cases/10_statistical_segments.yaml - Data shape:
entity_with_segments - Analytics type:
statistical