When History Misleads: Detecting Model-Dependent Counterfactual Claims

Why This Problem Matters

Counterfactual inference—asking “what if” questions and estimating causal effects—underpins prediction and policy advice in political science. Gary King and Langche Zeng show that when proposed counterfactuals lie far outside the observed data, conclusions from even well-specified statistical models can rest on speculation and untestable modeling assumptions rather than empirical evidence. This undermines confidence in many high-stakes claims about institutional change and international interventions.

How the Authors Tackle It

King and Zeng develop straightforward, easy-to-apply diagnostic methods that reveal when counterfactual claims are driven by model-dependence rather than by the data. Crucially, these procedures do not require performing sensitivity tests over pre-specified classes of alternative models. Instead, the diagnostics assess whether the data contain enough information to support the counterfactual inferences researchers are drawing.

What They Do in Practice

• Present diagnostic tests that quantify the degree to which conclusions depend on unverified modeling choices.
• Apply these diagnostics to two broad literatures in political science: studies of the effects of changes in the degree of democracy and separate analyses of UN peacebuilding efforts.

Key Findings

• Many published analyses in these literatures appear to rely more on modeling assumptions than on empirical information in the historical record.
• For some important research questions, historical data simply do not contain sufficient information to justify confident counterfactual claims.
• When an analysis fails the authors' tests, scholars can know that substantive results are sensitive to at least some non-empirical modeling choices.

Practical Tools and Implications

King and Zeng provide free software implementing all suggested diagnostics, enabling researchers to check the robustness of counterfactual claims before drawing policy-relevant conclusions. Their work encourages more cautious interpretation of “what if” arguments and offers concrete tools to improve transparency and credibility in causal inference across comparative and international political science.