Why Survey Vignettes Mislead: The Hidden Information Equivalence Problem

🧾 What This Paper Shows

Survey experiments often describe attributes in a hypothetical scenario to learn about those attributes' real-world effects. Those inferences rest on a frequently overlooked assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE fails when subjects, upon receiving information about one attribute, update beliefs about other attributes. For example, labeling a country 'a democracy' can change respondents' beliefs about its geographic location. When IE is violated, the measured effect of the manipulation need not equal the target quantity—the causal effect of beliefs about the focal attribute.

🔎 How the Argument Is Tested

• Formally defines the IE assumption and links it to the exclusion restriction from instrumental-variable analysis.
• Develops a method to predict likely IE violations before running an experiment (ex ante) and diagnostic placebo tests to detect violations after the fact (ex post).
• Empirically evaluates three strategies meant to achieve IE across four survey experiments, including an extension of a prominent study of the democratic peace.

📊 Key Findings

• Subjects routinely update background beliefs when given information about a focal attribute, producing IE violations.
• Because of these updates, treatment effects from vignette manipulations can diverge from the intended causal estimand (the effect of beliefs about the focal attribute).
• Placebo tests provide a viable ex-post diagnostic for IE violations.
• Three strategies for restoring IE were compared:
• Abstract encouragement (brief prompts to focus on focal attributes) is ineffective.
• Specifying background details in the vignette reduces imbalance on the specified details and on highly correlated background beliefs, but does not eliminate imbalance on other, less correlated beliefs.
• Embedding a natural experiment within the vignette can reduce imbalance across all background beliefs, but this approach introduces additional complications.

⚖️ Why It Matters

These results caution against assuming vignette treatments affect only the intended belief. The IE assumption is central to valid inference from survey experiments; diagnosing and addressing IE violations is necessary for trustworthy conclusions about how beliefs shape political outcomes, including claims about the democratic peace.