There is a number a student never sees and a promise behind it that they always feel. When a diagnostic says a student is “working at 62%,” what is that? Not a fact, like mass or length. An estimate — one draw from a distribution of scores that same student would produce across parallel tests, different moods, luckier and unluckier questions. The number is the estimate. The promise is about the rest of the distribution, and the promise can be kept well, kept badly, or — most often, in software — never made at all, leaving the student to assume the number was a fact.
This dispatch is about the machinery for keeping that promise. It is the least glamorous subject in this notebook and, in one specific sense, the most ethical: every other instrument we have built — the knowledge tracer, the adaptive selector, the spaced scheduler — consumes measurements as fuel, and a measurement that hides its noise quietly poisons everything downstream. An earlier dispatch asked how to choose the next question 14; this one asks what the answers add up to, and how much of that sum is signal.
True scores, and the first decomposition
Charles Spearman, in the 1904 paper that also introduced correlation to psychology, made the founding move: treat an observed score as the sum of two unobservable parts — a true score, the value a person would average over infinitely many independent testings, and an error, the part that varies from one testing to the next 1. The decomposition is a model, not a metaphysical claim; its power is that it makes “how noisy is this instrument?” an empirical question with an empirical answer.
The answer's arithmetic arrived within a few years, from an odd corner. William Brown and Spearman himself, publishing back-to-back in 1910, derived what is now the Spearman-Brown prophecy formula: if you know a test's reliability — the correlation between it and a hypothetical parallel form — you can compute exactly what a k-times longer test would achieve 23:
The curve in the figure below is that formula, drawn for three starting reliabilities, and its shape teaches two lessons at once. The steep part is early: doubling a mediocre ten-item test buys more than polishing a good one, which is why thin instruments should grow before they are refined. And the curve saturates: reliability approaches 1 asymptotically, so the last decimal place costs a fortune in items. Length is an instrument, but a blunt one — and, the formula's fine print adds, a multiplier of whatever it is given. Twenty copies of a bad item are twenty times as bad.
Alpha, and what it promises
Reliability still needed a number you could compute from a single administration — you cannot give every student a parallel form. Lee Cronbach's 1951 paper supplied it: coefficient alpha, the mean of all split-half reliabilities a test could produce, expressible as a ratio of variances 4. The figure above draws what alpha is made of: observed-score variance partitioned into the part a parallel test would share (signal) and the part it would not (noise); alpha is the first over the sum. It answers exactly one question — would this test agree with itself? — and answers nothing else, a point its everyday abuse constantly forgets.
The folk thresholds — 0.70 for research, 0.80 for low-stakes decisions, 0.90 for high stakes — trace largely to Nunnally's textbook, which proposed them as starting points and said so 9. They are conventions, not laws; a diagnostic that gates a student's study plan deserves more caution than a survey, and a ten-item check deserves less ceremony than either. What matters is that the number on the dashboard carries its noise with it. The conversion is the standard error of measurement, SEM = SD·√(1−r), and it turns reliability from a coefficient into something a student can see: a range around their score. At r = 0.85 the band is ±1.96·SEM ≈ ±0.76 SD; at r = 0.60 it is wider than the useful part of the scale. The hero plate draws the same student at three lengths of test — ten items and the band swallows the decision; forty-five and the band is finally narrower than the choice it informs.
The Rasch turn
Classical theory has a hidden dependency: every number it produces — reliability included — is entangled with the particular sample of students it was computed on. Georg Rasch, a Danish mathematician working on reading tests, proposed something stranger and better: models in which each item has a difficulty and each person an ability on one shared scale, such that the comparison of two people is independent of which items were used, and the comparison of two items independent of which students took them — specific objectivity, he called it 6. The Rasch model and its item-response-theory descendants rebuilt measurement against the instrument rather than the cohort 78.
For our purposes the decisive gift is the information function. Each item, in the logistic model, informs about ability with a curve of its own — an easy item is informative about weak students, a hard one about strong students, each peaking near its own difficulty. A test's information is the sum of its items', and the standard error at any ability is the reciprocal of the square root of that sum. The figure above draws three items and their total: the test measures some students well and others badly, and you can see where. This is the arithmetic behind the adaptive selector of the earlier dispatch — aim each next question where the current information curve is thinnest, and the same number of questions buys a sharper estimate 14. It is also why two students can sit the same adaptive diagnostic and receive bands of different widths: the honest instrument does not pretend its precision is uniform.
Validity is an argument
Reliability's twin is validity, and here the field spent decades chasing a definition before settling on something more honest than any definition: an argument. Samuel Messick reframed validity as a unified judgment about the interpretability of scores and the worth of their consequences 10. Michael Kane made it operational: a score supports an interpretation; the interpretation licenses a decision; the decision produces consequences — and every link in that chain demands its own evidence, which the test-maker has an obligation to assemble and defend 11. The Standards for Educational and Psychological Testing say the same in the register of law 15.
The chain is why we keep saying bands. A point score invites the interpretation “this is the student's ability,” which licenses decisions the evidence cannot carry. A range with an honest width invites “the evidence locates the student between here and here,” which licenses exactly as much decision as it should. When a diagnostic has too little evidence to speak — a topic with three answered questions, a bank still in review — the honest reading is a dash and a promise to keep listening, not a fabricated zero dressed as data. An instrument that will not say “I don't know yet” will eventually say something false with perfect confidence.
Where the model thumbs the scale
Three limits, stated as house rules.
The band is not the truth. SEM quantifies one kind of noise — retesting variance under the same instrument. It does not cover a student's morning, a misread stem, or the gap between “able to do this” and “did do this.” Ranges are honest, not sacred.
Reliability is not validity. A perfectly consistent instrument can be consistently measuring the wrong thing — alpha says nothing about what a score means, and the most reliable diagnostic in the world is worthless if its items sample the syllabus badly. Wilson's construct maps are the discipline here: define what is being measured before counting what was 13.
Precision is bought with time, and time is the student's. The √n law is unforgiving: halving a band costs fourfold the questions. The ethical instrument spends the student's patience where the decision needs the sharpness — and says so, rather than asking forty-five questions to decorate a dashboard.
What we read from it
EuraStudy's diagnostic reports what the theory tells an instrument to report. Readings come as ranges with honest widths, not points; a reading appears only where the evidence supports one, and the dash — not a zero, not a percentage — marks the ground not yet covered. The adaptive selector spends questions where the information curve is thin 14, so a student's time buys the most measurement it can. And no number on the report claims more than its band: where the platform's honesty law says a value must be real or absent, this is the theory that says why.
The reading from a century of measurement theory is threefold. Every score is two numbers — the estimate and its noise; showing one while hiding the other is the original sin of dashboards. Length is arithmetic — the prophecy formula prices every additional question before you spend it. And validity is a chain of arguments, not a badge — a score earns its use link by link, and the instrument that shows its noise is the only one whose links hold.
References
- 1.Spearman, C. (1904). “General intelligence,” objectively determined and measured. American Journal of Psychology, 15(2), 201–292.
- 2.Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322.
- 3.Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271–295.
- 4.Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
- 5.Lord, F. M., & Novick, M. R. (1968). Statistical Theories of Mental Test Scores (with contributions by A. Birnbaum). Reading, MA: Addison-Wesley.
- 6.Rasch, G. (1960). Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen: Danish Institute for Educational Research. (Expanded English edition, 1980, University of Chicago Press.)
- 7.Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for Psychologists. Mahwah, NJ: Erlbaum.
- 8.Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38–47.
- 9.Nunnally, J. C. (1978). Psychometric Theory (2nd ed.). New York: McGraw-Hill.
- 10.Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
- 11.Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
- 12.Bond, T. G., & Fox, C. M. (2007). Applying the Rasch Model: Fundamental Measurement in the Human Sciences (2nd ed.). Mahwah, NJ: Erlbaum.
- 13.Wilson, M. (2005). Constructing Measures: An Item Response Modeling Approach. Mahwah, NJ: Erlbaum.
- 14.Twenty Questions (2026). The Lab, EuraStudy. /research/twenty-questions — adaptive testing and item information, the selector that asks the revealing question.
- 15.American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. Washington, DC: AERA.