dcc_001 · Deployment Compose: LLM assigns correct intents to a 3-section report

Status: ✓ Passing · Last run: 2026-06-09T07:12:02 · Pipeline: deployment_compose · History: 1/1 runs passed (100%)

What was tested

Exercises understand_deployment_goal against a hand-crafted report with three sections of unambiguous intent:

1. kpi_overview — a KPI dashboard query (count by stage, conversion rate). Should be kept unchanged for the live refresh. 2. fit_model — calls sklearn.LogisticRegression.fit(X, y) on closed opportunities. Should be transformed to scoring in live mode. 3. methodology — static markdown describing the model. LLM has leeway here (keep or drop); not asserted.

Asserts the LLM returns: - a non-empty deployment goal, - per_section intents for at least the 3 input sections, - kpi_overview → "keep", - fit_model → "transform_score" (or "keep" if it explicitly flagged the model section — but transform_score is the right call given has_model=True).

What we planted in the data

• 1 entities
• Signal strength: 0.50 (sigmoid slope multiplier)
• has_model = True
• expected_intents = {'kpi_overview': 'keep', 'fit_model': 'transform_score'}

What the system did

• Model selected: DeploymentComposeRunner

What we checked — all assertions passed

How this could have gone wrong

(no assertion descriptions provided)

Why this case matters

This is the LLM call that decides what every deployment will actually do. If it picks "keep" on a training section, the daily refresh ships re-training code — the model fits a fresh model on every refresh, ignores the saved artifact, and customers see wildly unstable predictions. This case catches the regression before any user deploys.

Reproducing

# from auto_insight_api/
python -m validation.v2 run dcc_001 --pipeline deployment_compose -v

• Case config: validation/v2/cases/dcc_001_compose_intent.yaml
• Data shape: deployment_compose_happy
• Analytics type: deployment_compose

← back to validation report