🔎 Research Puzzle
Valid comparisons of group scores on additive measures like political knowledge scales require that, after controlling for an individual's overall level on the latent trait, the conditional probabilities of responding to each item are invariant across groups. If this measurement invariance fails, apparent differences between subgroups may reflect measurement artifacts rather than true knowledge gaps.
📊 What Was Analyzed and How
- Knowledge items drawn from the American National Election Studies (ANES) were examined using multi-group confirmatory factor analysis (MG-CFA).
- Differential item functioning (DIF) was assessed across a range of theoretically important grouping variables commonly used in political science research.
📌 Key Findings
- The knowledge scales used in recent research are not sufficiently invariant for valid comparisons across many commonly used grouping variables.
- It is possible to construct valid, invariant scales by selecting a subset of ANES items that meet invariance requirements.
- The impact of invariance (or lack thereof) is demonstrated by comparing substantive results from the valid subset-based scales with results from the original, invalid scales.
- DIF analysis identifies which items are most and least useful for building valid knowledge measures across groups.
- An application of the VTT indicates that ANES knowledge items are better conceptualized as effects of a latent variable (reflective indicators) rather than as causes or formative indicators.
⚖️ Why It Matters
- Comparisons of group knowledge levels in existing literature may be biased because many published scales lack demonstrated measurement invariance and were effectively "validated by fiat."
- Using invariant subsets of items changes substantive conclusions about subgroup knowledge differences and improves the validity of inference in studies of political knowledge.
🔍 Takeaway
- Measurement invariance must be established before comparing group scores on additive knowledge scales; doing so alters both scale construction and substantive conclusions about knowledge gaps across political and demographic groups.
