A clear question and practical guidance. Online survey panels have made sampling weights common for claims of representativeness, but uncertainty persists about when and how to use those weights in survey experiments. This paper offers concrete guidance—grounded in the Neyman–Rubin causal model—on estimators, their biases and variances, and when weighting helps or hurts.

🔬 Research Framework (Neyman–Rubin model)

• Uses the Neyman–Rubin potential-outcomes framework to evaluate simple, efficient estimators for survey experiment data.
• Derives analytic formulas for estimator bias and variance so researchers can compare the trade-offs of weighted and unweighted approaches.

🧪 Simulations and YouGov experiments

• Simulation studies compare estimator performance across realistic scenarios, including power and bias trade-offs.
• Real-world examples come from online experiments administered through YouGov, illustrating empirical behavior of estimators in practice.

📈 Key Findings

• For testing whether population treatment effects exist using high-quality, broadly representative samples recruited by top online survey firms, sample quantities that do not use weights are often sufficient.
• Sample average treatment effect (SATE) estimates typically do not differ substantially from their weighted counterparts but avoid the substantial loss of statistical power that accompanies weighting.
• When precise estimates of the population average treatment effect (PATE) are essential, poststratifying on survey weights and/or covariates highly correlated with outcomes is analytically shown to be a conservative choice.
• Simulations indicate substantial gains from poststratification in many settings, but limited evidence of such gains was found in the YouGov examples.

🔧 Practical recommendations for researchers

• Use unweighted SATE for testing the existence of effects in many high-quality online samples to preserve power.
• Reserve weighting or poststratification when the explicit goal is a precise PATE estimate, and prioritize covariates that are strongly correlated with outcomes.
• Consult the provided bias and variance formulas to anticipate trade-offs for specific designs.

💡 Why it matters

• The paper clarifies a common analytic dilemma in survey experiments, offering actionable rules that balance representativeness, bias, and statistical power so researchers can make informed choices about when to weight and how to do so conservatively.