vc_008 · Descriptive analytics — monthly client panel

Status: ✓ Passing · Last run: 2026-06-04T05:55:58 · Pipeline: analyze · History: 2/3 runs passed (67%)

What was tested

Same data shape vc_002 uses for ML (panel_snapshot — one row per (client, monthly snapshot)), but here the analyze_agent is asked structural questions about the panel itself: distinct entity count, number of snapshots, time span, marginal positive rate. Tests that the chat surface correctly introspects a panel — counting distinct entities (DISTINCT client_id) is meaningfully different from counting rows, and confusing them is a common analytical mistake the agent must not make.

What we planted in the data

• 100 entities
• 12 snapshots per entity at 30-day cadence
• Planted driver: usage_score
• 2 noise feature(s) (no relationship to outcome)
• Target positive rate ≈ 30%
• Signal strength: 0.60 (sigmoid slope multiplier)

What the system did

• Model selected: analyze_agent

What we checked — all assertions passed

How this could have gone wrong

• If tool_was_used failed: DISTINCT count on a panel — must touch the data, not infer.
• If answer_includes_number_near failed: Exact distinct-client count is 100. Tight tolerance — the answer must clearly identify 100 (not the 1200 row count, a common panel-confusion mistake).
• If tool_was_used failed: Counting distinct snapshot_date values; aggregation against data.
• If answer_includes_number_near failed: 12 monthly snapshots. The agent must not confuse this with the 1200 row count or the 100 client count.
• If tool_was_used failed: Rate aggregation — AVG(target) or COUNT(*)/COUNT WHERE target=1.
• If answer_includes_number_near failed: Marginal positive rate is calibrated to 30% (planted 0.30, realized ~35% under this seed). Expecting percentage form because agents naturally answer "35%" rather than "0.35"; tolerance is wide enough to cover sampling drift across seeds. Why % rather than decimal: the agent will report "35.2%" or "35%" — numbers like 35 dominate over fractional 0.35 in the reply, and answer_includes_number_near picks the closest number, which would be 35 not 0.35.
• If tool_was_used failed: MIN/MAX(snapshot_date) or a date-distribution from profile.
• If answer_mentions_fact failed: The panel starts at 2022-01-01 and runs ~11 months — the agent's reply must mention 2022. We don't pin the exact end-date because the agent's phrasing varies ("2022-11-27", "November 2022", etc.).

Why this case matters

Panels are where analytical chat agents go wrong most often. Common failures: confusing row count with entity count, reporting the POOLED positive rate when the per-snapshot rate was asked, missing the time dimension entirely. This case is the canary that the chat surface handles all three correctly on the simplest possible panel.

Reproducing

# from auto_insight_api/
python -m validation.v2 run vc_008 --pipeline analyze -v

• Case config: validation/v2/cases/08_descriptive_panel.yaml
• Data shape: panel_snapshot
• Analytics type: analytical_report

← back to validation report