🧠 Why guessing matters
Guessing on closed-ended knowledge items is common. Under likely-to-hold assumptions, the most common estimator of learning—the simple difference between pre- and post-process scores—is negatively biased when guessing occurs.
🔍 A latent-class fix that separates guessing from learning
- Introduces a latent class model of how people respond to knowledge questions that explicitly accounts for guessing-related error.
- Identifies the model using a mild and defensible assumption: people do not lose knowledge over short periods of time.
🧪 Simulation evidence across many item and process types
A Monte Carlo simulation over a broad range of informative processes and knowledge items finds:
- The simple difference score is negatively biased in the presence of guessing.
- The proposed latent class estimator is unbiased under the simulated conditions.
📣 Real-world test with Deliberative Polls
Applied to Deliberative Polls data, estimates of learning adjusted for guessing are about 13% higher than unadjusted difference scores. Accounting for guessing also eliminates the measured gender gap in learning and halves the pre-deliberation gender gap in political knowledge.
⚖️ Why it matters
Failing to adjust for guessing systematically underestimates learning and can create or exaggerate group disparities. The latent class approach offers a practical correction with a straightforward identifying assumption, improving the accuracy of learning measurement on closed-ended items.
