vc_006 · Descriptive analytics — segment questions
Status: ✓ Passing · Last run: 2026-06-04T05:55:25 · Pipeline: analyze
· History: 4/7 runs passed (57%)
What was tested
Customer-style table with categorical segments + numeric revenue + dates. The analyze_agent is asked three descriptive questions whose answers are planted facts about the synthetic data. Validates that the chat surface routes to the right tool (BigQuery for descriptive lookups), reads the data instead of guessing, and surfaces the correct value in its reply.
What we planted in the data
- 1,000 entities
- Signal strength: 0.50 (sigmoid slope multiplier)
What the system did
- Model selected:
analyze_agent
What we checked — all assertions passed
| Status | Assertion | Detail |
|---|---|---|
| ✓ | tool_was_used |
agent used query_bigquery in session up through 'q_segment_count' |
| ✓ | answer_mentions_fact |
all expected facts present in answer to 'q_segment_count' |
| ✓ | tool_was_used |
agent used query_bigquery in session up through 'q_segment_count' |
| ✓ | tool_was_used |
agent used query_bigquery in session up through 'q_segment_count' |
| ✓ | answer_mentions_fact |
all expected facts present in answer to 'q_segment_count' |
How this could have gone wrong
- If
tool_was_usedfailed: For a descriptive aggregation question, the agent should query BQ — not answer from prior context or hallucinate the answer. - If
answer_mentions_factfailed: Enterprise is the planted top segment by mean revenue. The agent's reply must name it. - If
tool_was_usedfailed: Total-revenue is a sum aggregation — must come from BQ, not a guess. - If
tool_was_usedfailed: Counting rows per segment is again a BQ aggregation. - If
answer_mentions_factfailed: SMB carries 40% of the synthetic population by construction — the agent should identify it as the largest segment.
Why this case matters
Validates the broader analytical-chat surface (the /analyze flow buyers see) on the simplest analytical question class — descriptive aggregations. If this case fails the failure is one of: (a) the agent didn't route to the BigQuery tool, (b) it answered without calling any tool, or (c) it reported the wrong segment. Each of those is a buyer-visible regression.
Reproducing
# from auto_insight_api/
python -m validation.v2 run vc_006 --pipeline analyze -v
- Case config:
validation/v2/cases/06_descriptive_segment.yaml - Data shape:
entity_with_segments - Analytics type:
analytical_report