dct_001 · Deployment Transform: LLM rewrites training section into live scoring
Status: ✓ Passing · Last run: 2026-06-09T06:56:45 · Pipeline: deployment_transform
· History: 1/1 runs passed (100%)
What was tested
Exercises deployment_composer.skills.meta.transform_section_for_live with intent="transform_score". The LLM (Claude Sonnet) receives a training-flavored Python block that calls LogisticRegression.fit(X, y) on labeled rows; it must return a rewrite that: - loads the saved model from the supplied GCS URI (joblib.load or pickle.load), - selects the COMPLEMENT of the training filter (rows that still need a prediction), - calls predict_proba / predict, NOT .fit(, - binds result to a pandas DataFrame with the predictions.
No GCS or BQ I/O — this is a pure LLM round-trip test. Catches prompt regressions, model-id drift, and AST-validator changes that silently reject good rewrites.
What we planted in the data
- 1 entities
- Signal strength: 0.50 (sigmoid slope multiplier)
What the system did
- Model selected:
DeploymentTransformRunner
What we checked — all assertions passed
| Status | Assertion | Detail |
|---|---|---|
| ✓ | transform_succeeded |
transformed_code=976 chars, intent_applied=transform_score |
| ✓ | transform_loads_model |
transformed code loads the model via joblib.load |
| ✓ | transform_inverted_training |
transformed code has no .fit( calls (training was inverted) |
| ✓ | transform_references_model_uri |
transformed code references model_uri (gs://auto-insight-api/validation_transform/dct_test/model.pkl) |
How this could have gone wrong
(no assertion descriptions provided)
Why this case matters
When a customer ships a deployment, this is the LLM call that decides what the daily refresh actually computes. If it stops producing valid scoring code (prompt regression, AST-rule change, model availability), every NEW deployment from that point becomes a ticking time bomb — the deploy succeeds but the daily refresh silently produces garbage. This case fires every time the suite runs.
Reproducing
# from auto_insight_api/
python -m validation.v2 run dct_001 --pipeline deployment_transform -v
- Case config:
validation/v2/cases/dct_001_transform_happy.yaml - Data shape:
deployment_transform_happy - Analytics type:
deployment_transform