vc_007 · Descriptive analytics — single-table entity data

Status: ✓ Passing · Last run: 2026-06-04T06:54:33 · Pipeline: analyze · History: 4/6 runs passed (67%)

What was tested

The simplest possible analytical-chat scenario: one row per entity, one planted feature, a binary target, no time dimension. The analyze_agent is asked structural questions (row count, target distribution) whose expected answers are known directly from the case's planted_truth parameters — no realized-random-draw dependence. Distinct from vc_006 because the questions and expected facts are declared in this YAML rather than computed by the generator. Proves the framework supports analytical_report cases on any data shape, not just shapes whose generator happens to author the questions.

What we planted in the data

• 1,500 entities
• Planted driver: usage_score
• 2 noise feature(s) (no relationship to outcome)
• Target positive rate ≈ 30%
• Signal strength: 0.60 (sigmoid slope multiplier)

What the system did

• Model selected: analyze_agent

What we checked — all assertions passed

How this could have gone wrong

• If tool_was_used failed: A row-count question — must be answered from the data, not hallucinated. Either query_bigquery (SELECT COUNT(*)) or profile_data (table profile carries row count) is valid.
• If answer_includes_number_near failed: Exact row count: agent's reply contains 1500 (allowing for comma formatting; tight tolerance because the row count is exact).
• If tool_was_used failed: Rate calculations require touching the data. Either an aggregating SQL query or the profile's distribution metadata is acceptable.
• If answer_includes_number_near failed: The realized positive rate is within sampling noise of 0.30 for n=1500. Lenient tolerance (20%) AND a list of equivalent expected forms (decimal 0.30 OR percentage 30) — surfaced by slow-lane drift detection: the agent reported "0.32" locally and "32%" in CI for identical prompts. Either is correct; the checker now accepts both.
• If tool_was_used failed: Schema questions: profile or a SELECT-with-LIMIT both work. The check is "did the agent look at the table", not "which SQL verb did it use".
• If answer_mentions_fact failed: The outcome column is literally named target. The agent's reply must surface this name.

Why this case matters

Without this case, every analytical_report test relies on the entity_with_segments shape — a richer but narrower scenario. Many real customers ship data closer to entity_static (one row per account, no segments, simple features). This case confirms the chat surface handles that baseline shape, and exercises the case-level questions: declaration path so future cases on other shapes (panel, multi-table, event) are unblocked.

Reproducing

# from auto_insight_api/
python -m validation.v2 run vc_007 --pipeline analyze -v

• Case config: validation/v2/cases/07_descriptive_entity_static.yaml
• Data shape: entity_static
• Analytics type: analytical_report

← back to validation report