Accounting for Guessing Raises Estimated Learning by 13% and Eliminates Gender Gap

🧠 Why guessing matters

Guessing on closed-ended knowledge items is common. Under likely-to-hold assumptions, the most common estimator of learning—the simple difference between pre- and post-process scores—is negatively biased when guessing occurs.

🔍 A latent-class fix that separates guessing from learning

• Introduces a latent class model of how people respond to knowledge questions that explicitly accounts for guessing-related error.
• Identifies the model using a mild and defensible assumption: people do not lose knowledge over short periods of time.

🧪 Simulation evidence across many item and process types

A Monte Carlo simulation over a broad range of informative processes and knowledge items finds:

• The simple difference score is negatively biased in the presence of guessing.
• The proposed latent class estimator is unbiased under the simulated conditions.

📣 Real-world test with Deliberative Polls

Applied to Deliberative Polls data, estimates of learning adjusted for guessing are about 13% higher than unadjusted difference scores. Accounting for guessing also eliminates the measured gender gap in learning and halves the pre-deliberation gender gap in political knowledge.

⚖️ Why it matters

Failing to adjust for guessing systematically underestimates learning and can create or exaggerate group disparities. The latent class approach offers a practical correction with a straightforward identifying assumption, improving the accuracy of learning measurement on closed-ended items.