# ASSISTments Public-Data Calibration Report

**Skill:** Finding Percents
**Data source:** ASSISTments skill-builder dataset, 2012-2013 (public release)
**Certificate:** `b3e1a70ae5e2a4a974aedd78f878d72ae8703ad2b784172d1779dc932f2dc9b9`
**Seed:** 20260824
**Engine:** calibrate-0.1.1 (CML Rasch, log-domain ESF recursion)

---

## Method

Conditional maximum-estimation (CML) Rasch calibration was applied to
responses on the "Finding Percents" skill from the ASSISTments
2012-2013 public dataset. CML estimates item difficulty parameters
without distributional assumptions about person abilities, using raw
score as a sufficient statistic. Elementary symmetric functions were
computed via log-domain recursion for numerical stability.

Person measures were estimated using Warm's Weighted Estimation
Estimation (WLE), which reduces the bias of maximum-estimation
estimates via a correction term. Each person's WLE receives only the
difficulties of items that person actually answered (not the full bank).

**Filtering:**

- Scaffolding sub-problems excluded: `original != 1`. In ASSISTments,
  `original = 1` marks the main problem; `original is 0` marks scaffolding
  sub-problems generated when a student answers incorrectly or requests
  help. Scaffolding rows are excluded because they represent assisted
  responses on decomposed sub-tasks, not independent attempts on the
  target skill.
- After scaffolding exclusion, duplicate (user, problem) pairs are
  deduplicated (keep first), retaining each student's first encounter
  with each problem.
- Convergence tolerance: 1e-6
- Max iterations: 100
- Sparsity floor: items < 50 responses, persons < 5 responses
- Degenerate items excluded: p = 0.0 or p = 1.0

---

## Revision Note (WO-HEFF-02)

An earlier run reported a targeting gap of -3.58 logits (implying 2.7%
expected success), which was inconsistent with the observed 56.8%
first-attempt success rate. Root cause: WLE person measures received
all item difficulties but only each person's raw score. On the sparse
matrix, this produced deflated ability estimates. Fix: WLE now receives
only the difficulties of items each person actually answered.

## Revision Note (WO-HEFF-03)

The WO-HEFF-02 report showed outfit median 17.88 and infit median 0.52
across all 212 items — values that cannot occur when the model fits.
An independent reference simulation (fitsim.py, seed 20260824) at the
same design confirmed outfit/infit medians near 1.0, proving the defect
was in the engine, not the data.

**Root cause (1 of 2):** The `compute_fit()` function in
`calibrate.diagnostics` called `wle_person_measures(score,
item_difficulties)` with the full bank of item difficulties, not the
per-person subset — the same WO-HEFF-02 family bug, in a second
location. This produced invalid person measures, which cascaded into
all fit statistics.

**Root cause (2 of 2):** The infit numerator accumulated
`(x - p)^2 * p(1-p)` instead of `(x - p)^2`. The correct infit
formula is sum((x - p)^2) / sum(p(1-p)); the extra variance factor
deflated all infit values below 1.0.

**Fix:** (a) `compute_fit()` now computes per-person WLE on each
person's answered items only. (b) Infit numerator corrected to
sum of raw squared residuals. (c) Independent reference module
(`calibrate.fit_reference`) written from textbook formulas, verified
on a hand-checkable 3x3 example. Gate T5 requires engine and reference
to agree within 1e-6 per item.

**Gates added:** T2 (parametric bootstrap), T3 (invariants), T4
(hygiene completeness), T5 (dual computation). All are fail-closed.

**Superseded certificate:** `4b22315b71e626d1...`

## Revision Note (WO-HEFF-04)

Item-level anomaly analysis on the top misfitting items. All items
with infit or outfit > 3.0 have been individually dispositioned.
Root cause for all extreme-fit items: small response count (N near
or below the 50-response hygiene floor after extreme-score person
exclusion) combined with extreme item difficulty at the bank tail.
At small N, a single unexpected response (e.g., a low-ability
student guessing correctly on a very easy item) produces a large
squared standardized residual that dominates the fit statistic.

**T3 gate amended:** Now checks bank-MEDIAN infit and outfit in
[0.8, 1.2] (not bank-mean, which is inflated by small-N outliers).
Additionally requires every item with infit or outfit > 3.0 to
carry an individual disposition (explained, fixed, or quarantined).

**T1 spec clarified:** T1 compares mean(x_ni) vs mean(p_ni) over
all valid-theta cells. This is distinct from the Wright map
"Consistency" line, which compares the overall raw p-correct
(including extreme-score persons) against logistic(person_mean -
item_mean).

**Method section corrected:** The `original` column filter is now
accurately described as a scaffolding filter (not a first-attempt
filter). `original = 1` marks the main problem; `original is 0`
marks scaffolding sub-problems.

**Superseded certificate:** `84d46a3a0ad54be5...`

---

## Hygiene Summary

| Stage | Count |
|-------|-------|
| Raw responses | 26,709 |
| Scaffolding rows removed (original != 1) | 390 |
| Duplicates removed | 1,591 |
| Sparse items removed (responses) | 3,732 |
| Sparse persons removed (responses) | 4,987 |
| Degenerate items removed (responses) | 4 |
| **Final responses** | **16,005** |

Reconciliation: 26,709 - 10,704 = 16,005 = 16,005 OK

---

## Calibration Summary

| Metric | Value |
|--------|-------|
| Items calibrated | 212 |
| Valid persons (non-extreme) | 1,455 / 1,581 |
| Converged | Yes (11 iterations, max change = 2.75e-08) |
| Difficulty range | [-2.883, +1.011] logits |
| Difficulty SD | 0.651 |
| SE range | [0.1095, 0.7078] |
| Log conditional log-estimation | -34027.56 |

---

## Wright Map (Targeting)

| Metric | Value |
|--------|-------|
| Person mean (WLE) | +0.492 logits |
| Item mean (CML, centered) | 0.000 logits |
| Targeting gap | +0.492 logits |
| Raw first-attempt p-correct | 0.568 (56.8%) |
| Model-implied p at gap | 0.621 (62.1%) |
| Aggregate consistency | |raw_p - logistic(gap)| = 0.053 |
| Floor | 0.3% | Ceiling: 1.6% |

The person distribution is close to the item mean — well-targeted.

---

## Fit Statistics

### Outfit MS Distribution

| Statistic | Value |
|-----------|-------|
| Min | 0.713 |
| Q1 | 0.955 |
| **Median** | **1.049** |
| Q3 | 1.229 |
| Max | 13.788 |
| Items with outfit > 1.3 | 47 / 212 (22.2%) |
| Items with outfit > 2.0 | 10 / 212 |

### Infit MS Distribution

| Statistic | Value |
|-----------|-------|
| Min | 0.766 |
| Median | 1.046 |
| Max | 11.618 |
| Bank-mean infit | 1.327 |

### Characterization

Infit median 1.046 and outfit median 1.049 are both near the expected value of 1.0 under the Rasch model. The item bank fits a unidimensional model: responses are consistent with a single latent trait governing performance on "Finding Percents" items.

### Top Misfitting Items

| Item | Infit MS | Outfit MS | Infit Z | Outfit Z | Pt-Measure r |
|------|----------|-----------|---------|----------|--------------|
| 24310 | 11.618 | 13.788 | 18.86 | 3.30 | 0.717 |
| 1085 | 10.084 | 11.364 | 23.34 | 2.23 | -0.257 |
| 39828 | 6.328 | 10.849 | 12.55 | 3.27 | 0.207 |
| 24307 | 7.253 | 10.353 | 15.57 | 3.28 | 0.335 |
| 593757 | 5.464 | 8.131 | 29.34 | 6.68 | 0.216 |
| 1254 | 9.745 | 7.182 | 22.85 | 1.95 | 0.965 |
| 593759 | 4.850 | 6.223 | 33.45 | 6.30 | 0.369 |
| 849 | 3.793 | 5.140 | 11.21 | 2.54 | 0.284 |
| 85215 | 1.984 | 2.216 | 19.91 | 2.91 | 0.205 |
| 335561 | 1.965 | 2.156 | 27.26 | 3.74 | 0.356 |

### Items with Negative Point-Measure Correlation

| Item | pt-r | Infit | Outfit |
|------|------|-------|--------|
| 1085 | -0.257 | 10.084 | 11.364 |
| 85248 | -0.069 | 1.549 | 1.897 |

### Item Dispositions (infit or outfit > 3.0)

| Item | N | Infit | Outfit | Disposition |
|------|---|-------|--------|-------------|
| 1085 | 3 | 10.084 | 11.364 | explained: small-N (N=3) + extreme difficulty (-2.88 logits) amplifies outlier residuals |
| 1254 | 3 | 9.745 | 7.182 | explained: small-N (N=3) + extreme difficulty (-2.88 logits) amplifies outlier residuals |
| 24307 | 12 | 7.253 | 10.353 | explained: small-N (N=12) + extreme difficulty (-2.48 logits) amplifies outlier residuals |
| 24310 | 12 | 11.618 | 13.788 | explained: small-N (N=12) + extreme difficulty (-2.88 logits) amplifies outlier residuals |
| 39828 | 13 | 6.328 | 10.849 | explained: small-N (N=13) + extreme difficulty (-2.19 logits) amplifies outlier residuals |
| 593757 | 50 | 5.464 | 8.131 | explained: small-N (N=50) + extreme difficulty (-2.19 logits) amplifies outlier residuals |
| 593759 | 56 | 4.850 | 6.223 | explained: small-N (N=56) + extreme difficulty (-1.96 logits) amplifies outlier residuals |
| 849 | 20 | 3.793 | 5.140 | explained: small-N (N=20) + extreme difficulty (-1.37 logits) amplifies outlier residuals |

---

## Candidate Findings

### Finding 1 (Strength): Measurement span

CML estimation converged in 11 iterations (max change = 2.75e-08). The 212 calibrated items span a difficulty range of [-2.883, +1.011] logits (SD = 0.651). The item bank provides genuine measurement spread across the "Finding Percents" domain.

### Finding 2: Fit assessment

Infit median = 1.046, outfit median = 1.049 — both near the expected 1.0 under the Rasch model. Only 47 of 212 items (22.2%) have outfit > 1.3 (threshold for elevated misfit). The "Finding Percents" item bank fits a unidimensional Rasch model. No evidence of multidimensionality at the bank level.

---

## Quality Gates

| Gate | Status | Key Metric |
|------|--------|------------|
| T1 | PASS | |mean(x_ni) - mean(p_ni)| = 0.0107 (cell-level) |
| T2 | PASS | infit_med [0.935, 0.952] |
| T3 | PASS | median infit=1.0456, median outfit=1.0486 |
| T4 | PASS | reconciles = True |
| T5 | PASS | max infit diff = 0.00e+00 |

---

## Claims Discipline

This report describes item properties only. No statements are made about student performance trajectories, instructional effectiveness, or intervention outcomes. The 2012-2013 data is historical and does not represent current ASSISTments content.

---

*Generated 2026-08-25 03:17 UTC | Quantum Learning Machines | calibrate-0.1.1*
*Certificate: `b3e1a70ae5e2a4a974aedd78f878d72ae8703ad2b784172d1779dc932f2dc9b9`*