Assertion Validation

vc_021 · Source declared finer than the modeling grain (aggregate, don't refuse)

Status: ○ No runs yet · Last run: — · Pipeline: —

What was tested

Reproduces the failure behind v2 runs 29ed9d7e14, d2099280 and 8a0e6a58 (2026-08-23): a claims table declared at (member_id, fill_date, claim_seq) with the question asked at MEMBER level. The source legitimately carries ~6 claims per member — those are rows to aggregate, not duplicates. Before the fix, _validate_entity_table_grain compared the SOURCE table against the modeling grain and halted:

"9414 rows but only 1465 distinct ENCRYPTED_MEMBER_ID values (7949 duplicate rows). The pipeline expects one row per entity. Please dedupe ... and retry."

That asks the customer to delete 84% of their data to model it. The pipeline already had the right machinery — unit_grid builds one row per entity, pred_grid crosses it with time_grid, and feature_plan's fans_out detector forces aggregation-only feature kinds — and already took that path for event-stream sources. The fix routes a source whose DECLARED key is strictly finer than the modeling key down that same path. Contrast with the two neighbouring cases, which must keep their behaviour: vc_014 dirty duplicate entity rows, NO declared key -> still halts vc_020 many-per-entity already classified event-stream -> already skips

What we planted in the data

What the system did

No run yet.

What we plan to check

Assertion What it checks
no_pipeline_errors The run must complete. This is the regression this case exists for: a halt here means we are again telling a customer to delete rows instead of aggregating them.
planted_feature_in_top_k_importance usage_score is a per-member latent echoed onto every claim, so it only survives if claims were aggregated to the member correctly. If the roll-up is wrong the planted driver washes out and this fails.
auc_in_band Band set from measurement, not from signal_strength. The first real run of this case scored CV AUC 0.976 (LightGBM, 51,615 rows, 50 of 95 features kept) with all three other assertions green and the planted driver at rank 1 — an honest result, not a leak. An earlier draft used [0.65, 0.97], reasoned from signal_strength=0.7 as though this were an entity-static case. It is not. The outcome is ABSORBING and the grid is SNAPSHOT-EXPANDED: once a member triggers, every later (member, date) row stays positive and is easy to classify, so a correct pipeline scores far higher here than the same signal strength would yield one-row-per-entity. The old ceiling failed a working pipeline by 0.006. The ceiling still earns its place. A column that DEFINES the outcome drives this shape to ~0.999, so 0.99 discriminates leakage from structure. Proper leak-guard is vc_003's job; this is a backstop. Below 0.90 means the per-member roll-up destroyed the signal, which is the failure this case is really watching for.
noise_features_below_planted Noise varies per claim, so a bad aggregation can manufacture spurious signal from it. Every noise feature must still rank below the planted driver.

How this could have gone wrong

Why this case matters

Claims, transactions, events and visits are the most common shapes customers arrive with, and the interesting question is almost always one level up: per member, per customer, per account. Refusing a finer-grained source is refusing the normal case — and the error told the user to destroy data that was correctly loaded, having been asked about the grain at load time and having answered correctly. This case is the floor that keeps the aggregate path in place.

Reproducing

# from auto_insight_api/
python -m validation.v2 run vc_021 --pipeline v2 -v

← back to validation report