Representative Samples Don't Guarantee Regression Estimates: A Surprising Flaw

Does regression produce representative estimates of causal effects? This question challenges conventional wisdom, even with seemingly ideal data. While unrepresentative samples can bias results, this problem persists in studies using large, representative datasets—like the American National Election Study (ANES). Conventional estimation methods via multiple regression still differentially weight each unit's contribution to the "effective sample," potentially yielding misleading findings.

This paper explores how these conventional estimates might produce nonrepresentative causal effects. It reveals that even with a representative population, standard regression techniques lack external validity justification compared to quasi-experimental or experimental approaches. To address this, we introduce multiple regression weights—allowing detailed analysis of the effective sample—and discuss alternative methods.

Key findings highlight specific conditions under which these alternatives might recover true average causal effects from a representative population. However, crucially, these necessary conditions cannot always be met in practice.