Turn Open-Ended Answers Into Continuous Attention Checks

Participants in online surveys and experiments are often inattentive, which can undermine substantive and causal inferences. Many practitioners therefore include multiple factual or instructional closed-ended manipulation checks to identify low-attention respondents. Closed-ended checks are binary (correct or incorrect), which makes them easier to guess and limits variation in measured attention.

📝 What This Paper Proposes

An automatic, standardized text-as-data approach that uses the text respondents provide in an open-ended manipulation check to measure attention. The method converts open-ended answers into an automated, continuous attention score that can be applied consistently across studies.

🔎 How Open-Ended Text Is Used to Measure Attention

• Uses respondents' free-text manipulation-check answers as the raw data source.
• Applies automated text-as-data techniques to produce a continuous attention measure rather than a binary pass/fail indicator.
• Reduces reliance on subjective, paid human coders by standardizing the coding process.

📈 Key Benefits and Diagnostics

• Continuous measurement: captures greater variation in attention across respondents compared with closed-ended checks.
• Robust automation: minimizes the cost and variability introduced by human coding of open-ended responses.
• Diagnostic tools: outlines procedures to diagnose how inattentive respondents affect overall results and how to estimate the average treatment effect among respondents likely to have received the treatment.

⚙️ Practical Tools and Implementation

Easy-to-use R software is provided to implement the open-ended manipulation-check workflow and the accompanying diagnostics, enabling researchers to adopt the approach without bespoke coding.

Why it matters: this standardized, automated text-based measurement improves attention assessment in survey experiments, clarifies the influence of low-attention respondents on inference, and offers practical code for immediate use.