Twenty Questions

A small piece of history hides in that logistic curve. The original model was the normal-ogive, written with the cumulative normal distribution (Thurstone in the 1920s; Lord in 1952); the logistic was adopted later because it is far easier to compute, and Birnbaum brought it into the theory in 1968 3. To keep the two interchangeable, item parameters are often multiplied by a scaling constant $D \approx 1.702$ — a value chosen (by Haley, in 1952) precisely so the logistic and normal curves never differ by more than about $0.01$. None of this is decoration: it is what lets a difficulty estimated under one model be read on the other’s scale. And it is the first sign of IRT’s real gift over older test theory — item properties and student ability live on the same scale, each estimable, in principle, independently of the particular sample of people or questions you happened to use 14.

What a question is worth

Here is the idea the whole article turns on. If you already have a rough estimate of a student’s ability, not every remaining question is equally useful — and you can say exactly how useful each one is. The measure is item information, and for a 2PL item it has a compact form:

$I(\theta) = a^{2}\,P(\theta)\,(1 - P(\theta))$

The name is not a metaphor. It is Fisher information — R. A. Fisher’s notion of the precision a single observation buys you about an unknown quantity, defined as the reciprocal of the variance with which you could estimate it 5. Read the formula and the strategy of adaptive testing falls out of it. The product $P(1-P)$ is largest when $P = 0.5$ — that is, when the item’s difficulty matches the student’s ability ($\theta = b$), the coin-flip question from the opening. Push the question too far above or below the student and $P$ races toward $1$ or $0$, the product collapses, and the item tells you almost nothing. The discrimination enters as $a^2$, so a sharp item is worth a great deal — its information peak rises as $a^2/4$ — but a glance at the curve shows that this tall peak is also narrow: a highly discriminating item is enormously informative about students near its difficulty and nearly useless about everyone else. (Guessing only makes this worse: a non-zero $c$ both lowers an item’s information and nudges its peak slightly above $b$, draining precision from exactly the low-ability students who are hardest to measure 5.)

So the question to ask next is not the hardest one, nor the one a syllabus lists first. It is the one whose difficulty sits closest to your current best guess of the student’s ability — the most informative question available. Everything else in adaptive testing is built on that single observation.

Precision you can see

Information has two more properties that turn it from an idea into an instrument. First, under the model’s assumption that answers are independent once ability is fixed, information simply adds up across the questions on a test:

$I(\theta) = \sum_i I_i(\theta)$

Second, that total converts directly into precision. The standard error of the ability estimate is

$\mathrm{SE}(\theta) = \frac{1}{\sqrt{I(\theta)}}$

so more information — better-targeted questions, or simply more of them — means a smaller error bar 45. This is the quiet revolution IRT works on the older classical test theory, which hands back a single reliability and a single standard error for the entire test, the same for a struggling student and a brilliant one. IRT instead makes precision a function of ability: a test can measure the middle of the range beautifully and the extremes poorly, and the test information curve shows you exactly where it is sharp and where it is blunt 4.

Now the engine is assembled. To drive the error bar down as fast as possible, spend each question where information is highest — which is to say, near the student’s ability — and watch the standard error fall with every well-aimed answer. That is the whole of adaptive testing in one sentence; the rest is making it work.

The loop that learns you

As an algorithm, computerized adaptive testing is a short loop 91011:

• Start. With no answers yet, seed the ability estimate at the population mean and choose a first item of middling difficulty.
• Select. Of the items not yet used, pick the one with the most information at the current estimate — the maximum-information rule.
• Administer and score the item.
• Re-estimate the ability from every answer so far.
• Stop when the standard error drops below a target (a variable-length test) or a fixed budget of items is spent; otherwise, loop back to select.

The re-estimation step hides a genuine subtlety, and it is where the cold start bites. The natural estimator, maximum likelihood, has no finite value while a student’s answers are all correct or all wrong — and at the very first question they always are 4. Adaptive tests therefore lean on Bayesian estimation, most often expected a posteriori (EAP): treat the population distribution as a prior and report the mean of the posterior after each answer. It is finite from the first item, needs no iteration, and was designed by Bock and Mislevy in 1982 for exactly this setting — running a live ability estimate on the slender computers of the day 7. A related caution shapes the early items: when the estimate is still far from the truth, the local information at a wrong estimate can point to the wrong questions, so methods that weigh information more globally — Chang and Ying’s use of Kullback–Leibler information — behave more steadily at the start 11.

Run the loop and the estimate marches toward the truth while its error bar contracts around it: a correct answer lifts the estimate, a wrong one lowers it, and each well-aimed question shaves more off the uncertainty than a randomly chosen one would. This is the source of the efficiency claim we began with — the test is confident about a student long before a fixed form of the same precision would have finished 8.

Where adaptivity has to be reined in

A test that only ever chased maximum information would be efficient and, in several ways, irresponsible. The interesting engineering of real systems is in the constraints layered on top of the simple rule.

The first is content. An exam must cover a blueprint — so many items of algebra, so many of geometry — and the most informative item at a given moment may be the wrong topic entirely. Constrained selection handles this by, in effect, assembling a complete test that satisfies every requirement and then administering its most informative item: the weighted-deviations method, and van der Linden’s elegant shadow-test approach, both keep the syllabus whole while still adapting 9.

The second is exposure and security. The maximum-information rule keeps reaching for the same handful of excellent items, which means those items are seen by almost everyone — and a question that everyone sees is a question that leaks. Exposure controls deliberately hold the best items back some fraction of the time (the Sympson–Hetter method 13), and a-stratified designs save the most discriminating items for late in the test, when the estimate is good enough to deserve them, spreading the load across the bank 12. The stakes are concrete: the Duolingo English Test reports a mean item-exposure rate near $0.1\%$, against the double-digit percentages typical of older operational adaptive tests — the difference between an item pool that stays secret and one that quietly ends up online 15.

The third, and the most important, is fairness. An item exhibits differential item functioning (DIF) when students of the same ability but different groups have different probabilities of answering it correctly — when its characteristic curve depends on who is taking it, beyond the ability it is meant to measure. That is a validity problem before it is a fairness problem, and it is screened with care, by methods from the Mantel–Haenszel procedure to comparing the area between two groups’ curves 141. Adaptive delivery inherits the issue wholesale: a biased item is biased no matter how much information it carries, and an efficient test that quietly disadvantages a group has optimised the wrong thing.

It is worth saying plainly that adaptivity is a choice, not an upgrade. Some of the highest-stakes computerized exams are deliberately fixed-length — the United States medical licensing examination among them — while others, like the NCLEX nursing examination, stop only once the test is statistically confident a candidate is above or below the passing standard. The right design depends on whether you are placing someone on a scale or simply deciding which side of a line they fall.

When the test learns to write itself

The classical pipeline has an expensive prerequisite hiding in plain sight: before an item can be used, its difficulty, discrimination and guessing must be calibrated by piloting it on hundreds of students. That is slow, costly, and a poor fit for a world that wants fresh questions constantly. The most striking recent development attacks exactly this. The Duolingo English Test uses machine learning to predict an item’s difficulty directly from its text — calibrating questions before a single examinee has seen them — and then delivers them adaptively; its machine-learned proficiency estimates line up with the classical IRT ability estimates at a rank correlation around $0.96$ 15. Other work reaches deeper into the loop, learning the selection policy itself from data rather than trusting the fixed maximum-information rule: BOBCAT casts adaptive testing as a bilevel optimisation and learns a data-driven question-picker that shortens tests further 16.

There is also a quieter connection, and it leads straight back to a companion idea. Item response theory draws a snapshot — a student’s ability fixed at the moment of the test. Knowledge tracing, the subject of an earlier dispatch, draws the moving picture — competence changing as a student learns 17. They are two views of the same latent thing, and recent work makes the bridge explicit, reading deep knowledge tracing as a kind of dynamic, multidimensional item response model 18. Measurement and learning, it turns out, are not separate theories so much as the still photograph and the film of one quantity we can never see directly.

What we read from it

EuraStudy is built on a reviewed bank of roughly three thousand exam-style questions, each tagged to a topic — the raw material any of this machinery needs. The IRT lens is, more than anything, a discipline about what a question is worth. A question has no fixed value; its worth depends entirely on who is answering it. The most useful question is rarely the hardest or the easiest — it is the one balanced at the edge of what a student can currently do. And precision is not a single number stamped on a test but something you can point at, target, and stop chasing once you have enough of it.

The honest reading is that the method and its limits are inseparable. A shorter test is only a better test when it is pointed in the right direction; an efficient one is worthless if it is unfair, narrow, or insecure. The aim of measuring a student well was never to ask the most questions. It is to ask the right ones — and, like a good player of twenty questions, to know exactly when you have heard enough to stop.