🔎 Why run identical protocols across different modes
Experiments should be designed so that experimental measurement error can be detected. Implementing identical protocols across diverse modes makes it possible to see whether apparent treatment effect heterogeneity is driven by substantive differences or by measurement problems.
🧭 How heterogeneous mode effects are estimated
- Iterative nonparametric estimation techniques are recommended to assess the magnitude and pattern of heterogeneous treatment effects across modes.
- These methods allow flexible, data-driven comparisons without imposing restrictive parametric assumptions.
🛠️ Diagnostics to tell signal from measurement error
- Measurement metrics embedded in experiments: include internal checks and outcome-quality metrics inside a single experiment to flag mode-related response differences.
- Measurement experiments: run dedicated experiments that vary mode to directly test whether observed heterogeneity reflects measurement error rather than true treatment heterogeneity.
🔬 Empirical illustration — Four identical interactive experiments
- Four implementations of the same interactive protocol were run to demonstrate the approach: a physical lab; online with subjects from the CESS lab subject pool; online with an online subject pool; and online with MTurk workers.
🌍 Follow-up measurement experiment in India
- A measurement experiment was implemented in India comparing CESS Online subjects and MTurk workers to further assess whether mode differences reflect measurement error.
📌 Why this matters
- Combining identical protocols across modes, iterative nonparametric estimation, and the two diagnostic strategies provides a practical toolkit to detect and diagnose experimental measurement error.
- These tools help researchers distinguish genuine heterogeneous treatment effects from artifacts of how an experiment is administered.
