How to use these tools as a measurement professional

What the paper numbers are

Exact agreement is percent agreement with the modal level per sample. Adjacent agreement adds scores one level away. Neither corrects for chance, neither weights disagreements beyond one level by distance, and both are computed on small grids, commonly 12 to 36 scores. The protocol chose them because a team can compute them by hand in two minutes and read them without training. Treat them as a monitoring signal, and treat a change of a few points in a single session as noise.

Getting more from the same session

The scoring grid holds everything a stronger analysis needs, provided the school keeps a keyed copy under its own control. The shared log deliberately omits scorer identity. A keyed grid with scorer pseudonyms, kept by the school and never forwarded, does not.

• Chance corrected agreement: quadratic weighted kappa or Krippendorff’s alpha (ordinal) on the same grid. With four to six samples per session, pool sessions on the same target before interpreting either.
• Rater effects: once a target has accumulated roughly 30 samples and a stable scorer pool, a many facet Rasch model separates rater severity from sample difficulty and scale step structure. Below that, report the descriptives and say so.
• Drift: plot exact and adjacent agreement per session on the same target. A control chart with limits from the first three sessions is enough to flag a shift.
• Scale separation: the distribution of modal levels across a semester shows whether the middle levels are being used. Two starred ties in one grid is the hand version of the same finding.

Checking the credit rule

Section 8 of the template is a cohort follow up. The unit is a student marked proficient on a target under a stated rule. The outcome is whether the next fresh demonstration on that target met the level. Record the support type that preceded the qualifying demonstration. With a few hundred decisions across a school year, the proportion holding by support type answers, in classroom data, the question the preregistered tutoring log analysis raised. If you run this, preregister the analysis. The registration at osf.io/b34vx is a usable pattern.

YardStick today

YardStick is QLM’s verification layer: an open evidence specification and an Apache 2.0 verifier SDK, published at play.quantumlearningmachines.com/developer. A measurement trail encoded in the specification can be verified by a third party without trusting QLM’s computation. Today the specification and SDK cover evidence trails produced by QLM’s instruments. [Kumar: confirm scope before publishing]

Planned, not shipped

The engineering spec that accompanies these tools asks for the following. None of it exists yet, and this section will change when it does.

• A support received field and a credit rule identifier on every mastery claim in the evidence specification, so a claim can be checked against the rule that produced it.
• A log import for the scoring sheet’s CSV, so a school’s hand recorded sessions carry forward.
• YardStick Calibrate: chance corrected agreement, rater facets where the data allow, drift across sessions, and scale separation, computed from imported logs and keyed grids, with scorer identity protected in every export by default. [Kumar: confirm the name before publishing]

If you want to help

The log schema and the credit rule schema are in the engineering spec and will be published under the same license as the tools. Comments on the field definitions are welcome before they freeze: hello@quantumlearningmachines.com.