Validation report

Three test suites cover the platform end-to-end. The industry suite verifies the platform can solve canonical ML problems (churn, hospitalization, fraud, …) end-to-end on multi-table data. The deployment suite verifies the second leg of the ML lifecycle — scheduled re-scoring stays correct under drift, idempotency, and operational failure modes. The regression suite locks in fixes to specific bugs we've shipped — every fix gets a synthetic case that captures the failure mode so it can't quietly come back.

New to this page? Read the methodology first.

At a glance

• Industry suite: 8/8 problems passing on their latest run
• Deployment suite: 21/21 failure modes guarded on their latest run
• Canonical suite: 3/3 absolute-truth claims locked on their latest run
• Regression suite: 19/21 cases passing on their latest run (coverage: 21/112 matrix cells, 18%) — 1 failing, 1 not yet run
• 171 total runs recorded across both suites
• Last refreshed: 2026-08-24T17:43:52

Industry suite

Each row is a canonical ML problem an industry buyer would bring. The case loads a realistic 3-6 table dataset for that problem, runs the full pipeline end-to-end, and asserts on process integrity (no errors, ≥40% of generated features survive dry-run, ≥3 source tables contribute features, AUC at least meaningfully above random).

Canonical suite

The small set of claims where we're not asserting a structural property — we're asserting an EXACT value. Each case has a hand-recorded answer key locked in validation/v2/cases/canonical_keys/; the assertion compares the pipeline's actual output to the locked value within tolerance. If anything in the pipeline changes its computation — model retrain, feature engineering, library upgrade — the canonical case fires. Re-recording the answer key is a deliberate act, like updating a snapshot test.

Deployment suite

Training is only half of the ML lifecycle. The other half is scheduled re-scoring: a trained model wakes up on a cadence, reads new rows, writes predictions, and must NOT silently corrupt history. The suite is layered — each lane catches regressions the lane below it can't see:

• Contract (dc_*) — logical contract, synthetic data, in-process sklearn. Cheap and hermetic.
• End-to-end (dce_*) — real GCS round-trip + real BigQuery + real execute_live_refresh. Catches GCS auth, pickle format drift, runtime-globals-dict regressions.
• HTTP handler (dch_*) — drives handle_render_live end-to-end (Firestore round-trip + re_render_html LLM). Includes happy path + the error matrix (404 missing project, 400 no spec, 400 empty sections).
• Compose intent LLM (dcc_*) — exercises understand_deployment_goal (the LLM that decides per-section keep / transform_score / drop / augment at deploy time). Catches intent-classifier regressions before they ship to every new deployment.
• Transform LLM (dct_*) — exercises transform_section_for_live (the LLM that rewrites a training section into a scoring section at deploy time). Catches prompt or AST-validator regressions.
• Full lifecycle (dcl_*) — the cherry on top: train → compose → transform → deploy → fire → read live_artifact, all real, no mocks. Slow (~1 min) and costly (~$0.05/run) but the only test that proves integration.

Cloud Run Job creation + Cloud Scheduler firing are NOT in this suite — those are deploy-time orchestration that GCP guarantees once deploy_to_cloud returns a valid job ID. The failure modes that actually affect daily refresh live inside the job body, which dce_* / dch_* cover.

Contract (dc_*)

Each row is a failure mode of the deployment contract — what any future scheduled-rescoring runner must satisfy. Runs against synthetic data with in-process sklearn; cheap, hermetic, fast.

End-to-end (dce_*)

Same failure modes, but exercising the REAL production path: joblib-pickle a trained model to a real GCS bucket, upload source data to a real BigQuery table, invoke deployment_composer.skills.meta.execute_live_refresh.run_sections_capture_outputs. Slower (~10s/case) but this is the only lane that catches GCS auth issues, pickle format drift, or runtime-globals-dict regressions.

HTTP handler (dch_*)

Drives deployment_composer_api.handle_render_live against a real Firestore project. Covers the full HTTP-handler contract: Firestore read of deployment_spec, invocation of execute_live_refresh (which calls the re_render_html Sonnet LLM), and the write-back of live_artifact.html to Firestore. ~40s/case — the LLM is the long pole.

Compose intent LLM (dcc_*)

Pure-LLM round-trip against deployment_composer.skills.meta.understand_deployment_goal. Hands the LLM 3 sections with unambiguous intents (a KPI dashboard, a .fit() training block, a methodology blurb); asserts the LLM returns 'keep' on the dashboard and 'transform_score' on the training section. ~4s/case.

Transform LLM (dct_*)

Pure-LLM round-trip against deployment_composer.skills.meta.transform_section_for_live. Sends a training-flavored section (calls .fit()); asserts the LLM returns a rewrite that loads the saved model, doesn't fit anything, and references the literal model_uri. Catches prompt regressions before they corrupt every new deployment.

Orchestration (dco_*)

Real Cloud Scheduler create → verify config → teardown. Proves the contract for deploy_to_cloud: given a DeploymentSpec, the provisioner must create a scheduler entry with the right cron, timezone, HTTP target, body, and HMAC header — and teardown must cleanly remove it. ~3s/case. The contract is implementation-agnostic; any future rewrite has to pass these same assertions.

Full lifecycle (dcl_*)

End-to-end integration test: train a model, upload to GCS, upload source to BigQuery, run the composer (LLM intents), run the transform (LLM rewrites), persist DeploymentSpec to Firestore, fire handle_render_live (which calls execute_live_refresh and re_render_html), then read live_artifact back. The only test that proves all layers integrate. Slow (~1 min) and uses real Anthropic + real GCS + real BQ + real Firestore.

Regression suite

Every shipped fix has a case here that captures the failure mode it addressed — a tight regression guard. The matrix below shows which (data shape × analytics type) combinations are covered; · cells are honest gaps, not failures.

Coverage matrix

Recent failures

Cases that did not pass on their latest run. Every failure is disclosed here, not silently retried.

vc_020 — Event-stream entity table (Forj retention shape) (regression suite)

• AuthenticationError: Error code: 401 - {'type': 'error', 'error': {'type': 'authentication_error', 'message': 'API key is invalid.'}, 'request_id': None}
• Case detail →

All cases

See methodology.md for how cases are constructed, what assertion kinds mean, and how failures are disclosed.