# ASSISTments Public-Data Calibration Report

**Skill:** Ordering Fractions
**Data source:** ASSISTments skill-builder dataset, 2012-2013 (public release)
**Certificate:** `c8d6dfeb2121208cb2edb18b4faa623c6fe6b31b74120abaebbaf1c4596117eb`
**Seed:** 20260824
**Engine:** calibrate-0.1.1 (CML Rasch, log-domain ESF recursion)

---

## Method

Conditional maximum-estimation (CML) Rasch calibration was applied to
responses on the "Ordering Fractions" skill from the ASSISTments
2012-2013 public dataset. CML estimates item difficulty parameters
without distributional assumptions about person abilities, using raw
score as a sufficient statistic. Elementary symmetric functions were
computed via log-domain recursion for numerical stability.

Person measures were estimated using Warm's Weighted Estimation
Estimation (WLE), which reduces the bias of maximum-estimation
estimates via a correction term. Each person's WLE receives only the
difficulties of items that person actually answered (not the full bank).

**Filtering:**

- Scaffolding sub-problems excluded: `original != 1`. In ASSISTments,
  `original = 1` marks the main problem; `original is 0` marks scaffolding
  sub-problems generated when a student answers incorrectly or requests
  help. Scaffolding rows are excluded because they represent assisted
  responses on decomposed sub-tasks, not independent attempts on the
  target skill.
- After scaffolding exclusion, duplicate (user, problem) pairs are
  deduplicated (keep first), retaining each student's first encounter
  with each problem.
- Convergence tolerance: 1e-6
- Max iterations: 100
- Sparsity floor: items < 50 responses, persons < 5 responses
- Degenerate items excluded: p = 0.0 or p = 1.0

---

## Revision Note (WO-HEFF-02)

An earlier run reported a targeting gap of -3.58 logits (implying 2.7%
expected success), which was inconsistent with the observed 56.8%
first-attempt success rate. Root cause: WLE person measures received
all item difficulties but only each person's raw score. On the sparse
matrix, this produced deflated ability estimates. Fix: WLE now receives
only the difficulties of items each person actually answered.

## Revision Note (WO-HEFF-03)

The WO-HEFF-02 report showed outfit median 17.88 and infit median 0.52
across all 212 items — values that cannot occur when the model fits.
An independent reference simulation (fitsim.py, seed 20260824) at the
same design confirmed outfit/infit medians near 1.0, proving the defect
was in the engine, not the data.

**Root cause (1 of 2):** The `compute_fit()` function in
`calibrate.diagnostics` called `wle_person_measures(score,
item_difficulties)` with the full bank of item difficulties, not the
per-person subset — the same WO-HEFF-02 family bug, in a second
location. This produced invalid person measures, which cascaded into
all fit statistics.

**Root cause (2 of 2):** The infit numerator accumulated
`(x - p)^2 * p(1-p)` instead of `(x - p)^2`. The correct infit
formula is sum((x - p)^2) / sum(p(1-p)); the extra variance factor
deflated all infit values below 1.0.

**Fix:** (a) `compute_fit()` now computes per-person WLE on each
person's answered items only. (b) Infit numerator corrected to
sum of raw squared residuals. (c) Independent reference module
(`calibrate.fit_reference`) written from textbook formulas, verified
on a hand-checkable 3x3 example. Gate T5 requires engine and reference
to agree within 1e-6 per item.

**Gates added:** T2 (parametric bootstrap), T3 (invariants), T4
(hygiene completeness), T5 (dual computation). All are fail-closed.

**Superseded certificate:** `02c74ba6ad9645fb...`

## Revision Note (WO-HEFF-04)

Item-level anomaly analysis on the top misfitting items. All items
with infit or outfit > 3.0 have been individually dispositioned.
Root cause for all extreme-fit items: small response count (N near
or below the 50-response hygiene floor after extreme-score person
exclusion) combined with extreme item difficulty at the bank tail.
At small N, a single unexpected response (e.g., a low-ability
student guessing correctly on a very easy item) produces a large
squared standardized residual that dominates the fit statistic.

**T3 gate amended:** Now checks bank-MEDIAN infit and outfit in
[0.8, 1.2] (not bank-mean, which is inflated by small-N outliers).
Additionally requires every item with infit or outfit > 3.0 to
carry an individual disposition (explained, fixed, or quarantined).

**T1 spec clarified:** T1 compares mean(x_ni) vs mean(p_ni) over
all valid-theta cells. This is distinct from the Wright map
"Consistency" line, which compares the overall raw p-correct
(including extreme-score persons) against logistic(person_mean -
item_mean).

**Method section corrected:** The `original` column filter is now
accurately described as a scaffolding filter (not a first-attempt
filter). `original = 1` marks the main problem; `original is 0`
marks scaffolding sub-problems.

**Superseded certificate:** `8a1e7b593e08edc6...`

---

## Hygiene Summary

| Stage | Count |
|-------|-------|
| Raw responses | 22,509 |
| Scaffolding rows removed (original != 1) | 0 |
| Duplicates removed | 2,525 |
| Sparse items removed (responses) | 8,535 |
| Sparse persons removed (responses) | 3,347 |
| Degenerate items removed (responses) | 478 |
| **Final responses** | **7,624** |

Reconciliation: 22,509 - 14,885 = 7,624 = 7,624 OK

---

## Calibration Summary

| Metric | Value |
|--------|-------|
| Items calibrated | 71 |
| Valid persons (non-extreme) | 537 / 857 |
| Converged | Yes (9 iterations, max change = 8.89e-09) |
| Difficulty range | [-1.803, +1.066] logits |
| Difficulty SD | 0.608 |
| SE range | [0.1114, 0.3576] |
| Log conditional log-estimation | -9742.50 |

---

## Wright Map (Targeting)

| Metric | Value |
|--------|-------|
| Person mean (WLE) | +0.882 logits |
| Item mean (CML, centered) | 0.000 logits |
| Targeting gap | +0.882 logits |
| Raw first-attempt p-correct | 0.785 (78.5%) |
| Model-implied p at gap | 0.707 (70.7%) |
| Aggregate consistency | |raw_p - logistic(gap)| = 0.078 |
| Floor | 0.4% | Ceiling: 5.2% |

The person distribution is 0.88 logits above the item mean. The arriving student population slightly outperforms the item bank's center of mass — a modest positive targeting gap common in tutoring systems.

---

## Fit Statistics

### Outfit MS Distribution

| Statistic | Value |
|-----------|-------|
| Min | 0.423 |
| Q1 | 0.779 |
| **Median** | **0.911** |
| Q3 | 1.070 |
| Max | 6.072 |
| Items with outfit > 1.3 | 13 / 71 (18.3%) |
| Items with outfit > 2.0 | 8 / 71 |

### Infit MS Distribution

| Statistic | Value |
|-----------|-------|
| Min | 0.555 |
| Median | 0.932 |
| Max | 5.213 |
| Bank-mean infit | 1.137 |

### Characterization

Infit median 0.932 and outfit median 0.911 are both near the expected value of 1.0 under the Rasch model. The item bank fits a unidimensional model: responses are consistent with a single latent trait governing performance on "Ordering Fractions" items.

### Top Misfitting Items

| Item | Infit MS | Outfit MS | Infit Z | Outfit Z | Pt-Measure r |
|------|----------|-----------|---------|----------|--------------|
| 66284 | 5.213 | 6.072 | 12.68 | 2.46 | 0.293 |
| 66216 | 2.655 | 3.989 | 7.24 | 2.20 | -0.002 |
| 208642 | 2.645 | 3.745 | 10.84 | 2.79 | 0.126 |
| 65608 | 2.787 | 3.645 | 7.67 | 1.97 | 0.104 |
| 65677 | 2.457 | 2.624 | 13.47 | 2.32 | 0.404 |
| 524678 | 2.335 | 2.477 | 23.16 | 3.27 | 0.489 |
| 524680 | 1.932 | 2.064 | 19.69 | 2.69 | 0.391 |
| 66282 | 1.725 | 2.021 | 11.03 | 2.38 | 0.139 |
| 97836 | 1.552 | 1.833 | 13.49 | 2.69 | 0.249 |
| 66271 | 0.555 | 0.423 | -10.15 | -2.07 | 0.491 |

### Items with Negative Point-Measure Correlation

| Item | pt-r | Infit | Outfit |
|------|------|-------|--------|
| 66216 | -0.002 | 2.655 | 3.989 |

### Item Dispositions (infit or outfit > 3.0)

| Item | N | Infit | Outfit | Disposition |
|------|---|-------|--------|-------------|
| 208642 | 23 | 2.645 | 3.745 | explained: small-N (N=23) + extreme difficulty (-1.57 logits) amplifies outlier residuals |
| 65608 | 26 | 2.787 | 3.645 | explained: small-N (N=26) + extreme difficulty (-1.39 logits) amplifies outlier residuals |
| 66216 | 25 | 2.655 | 3.989 | explained: small-N (N=25) + extreme difficulty (-1.30 logits) amplifies outlier residuals |
| 66284 | 19 | 5.213 | 6.072 | explained: small-N (N=19) + extreme difficulty (-1.80 logits) amplifies outlier residuals |

---

## Candidate Findings

### Finding 1 (Strength): Measurement span

CML estimation converged in 9 iterations (max change = 8.89e-09). The 71 calibrated items span a difficulty range of [-1.803, +1.066] logits (SD = 0.608). The item bank provides genuine measurement spread across the "Ordering Fractions" domain.

### Finding 2: Fit assessment

Infit median = 0.932, outfit median = 0.911 — both near the expected 1.0 under the Rasch model. Only 13 of 71 items (18.3%) have outfit > 1.3 (threshold for elevated misfit). The "Ordering Fractions" item bank fits a unidimensional Rasch model. No evidence of multidimensionality at the bank level.

---

## Quality Gates

| Gate | Status | Key Metric |
|------|--------|------------|
| T1 | PASS | |mean(x_ni) - mean(p_ni)| = 0.0270 (cell-level) |
| T2 | PASS | infit_med [0.897, 0.937] |
| T3 | PASS | median infit=0.9322, median outfit=0.9106 |
| T4 | PASS | reconciles = True |
| T5 | PASS | max infit diff = 0.00e+00 |

---

## Claims Discipline

This report describes item properties only. No statements are made about student performance trajectories, instructional effectiveness, or intervention outcomes. The 2012-2013 data is historical and does not represent current ASSISTments content.

---

*Generated 2026-08-25 03:17 UTC | Quantum Learning Machines | calibrate-0.1.1*
*Certificate: `c8d6dfeb2121208cb2edb18b4faa623c6fe6b31b74120abaebbaf1c4596117eb`*