Three patterns that expose broken simulations. Distilled from real audit failures during development. Apply them to any simulation-based validity claim.
What it catches: shared random-number streams across cells that should be independent. If a statistic is identical to 3+ decimals across cells that differ on a driving factor, the randomness is not independent — the cells are scaled copies of one draw.
When we caught it: round 7. Session-effect eta had SD ratio 1.037 in all four pilot cells. Four independent draws landing on the same ratio to four decimals has probability ~10^-6. The fix: derive each cell's eta stream from a cell-specific seed.
FVR across 36 cells (should vary):
How to apply: take any per-cell rate from a vendor's simulation study. Round to 3 decimals. If more than 2 cells that differ on a configured factor share the same value, the randomness is suspect.
What it catches: an engine whose prior is not actually plumbed into its inference. If sweeping the prior across {0.50, 0.70, 0.85} produces identical posteriors, the prior is dead code and the engine's calibration claims are vacuous.
When we caught it: round 6. The engine's session-1 correct-rate proxy was giving it P(mastery)=0.926 in sparse cells while BKT sat at P(L0)=0.40. The +0.043 AUC advantage was information asymmetry, not architecture.
Key insight: AUC is rank-invariant — a uniform prior shift doesn't change AUC even though it changes posteriors. Use ECE (calibration error), not AUC, to detect prior plumbing.
How to apply: run the candidate engine at three different priors on identical evidence. If ECE doesn't change, the prior is not connected. If AUC changes but ECE doesn't, something else is wrong.
What it catches: an engine that receives a better prior than its comparator, making the comparison measure information access rather than inference quality. The signature: sensitivity near 1.0 with specificity near 0 (or vice versa) — the engine is riding its prior instead of learning from evidence.
When we caught it: round 8. The engine got P(mastery)=0.926 from session-1 correct rate; BKT got P(L0)=0.40 (uninformed). Engine sensitivity 0.999 / specificity 0.056 — it was calling everything closed from the prior alone. The fix: symmetric prior invariant (Spec v0.2 Section 6.0).
How to apply: check the sensitivity/specificity pair for each estimator. If one is near 1.0 and the other near 0, the estimator is not discriminating — it's just following its prior. Then check whether the comparators received the same prior.
Tempered minus untempered ECE across all 36 cells. Negative = tempering helps. Pre-registered threshold: negative in at least 33 of 36.
0 of 0 cells show negative delta (tempering helps).