🔎 The Problem
Data sets that quantify social-scientific concepts often rely on multiple experts to code latent, ordinal concepts. Reporting the simple average across experts is common practice, but experts can differ in both their reliability and how they interpret rating scales. These differences make the mean a potentially biased or inaccurate summary.
📚 Data Sources: V-Dem and Realistic Simulations
- Cross-national expert-coded variables drawn from the Varieties of Democracy (V-Dem) data set.
- Ecologically motivated simulated panel data designed to mimic realistic patterns of expert disagreement.
🛠️ How Models Were Compared
A range of item-response theory (IRT) models was evaluated against the standard practice of reporting average expert codes. Comparisons focused on the ability of each approach to recover underlying latent concepts when experts vary in scale interpretation or reliability, and when differential item functioning (DIF) is present.
🧾 Key Findings
- IRT models outperform simple averages when experts vary in reliability.
- IRT approaches perform better than averages when experts exhibit differential item functioning (DIF).
- IRT models remain generally robust even when simulations include no DIF or no variation in expert reliability.
⚖️ Why It Matters
Item-response theory offers an intuitive, practical way to account for different expert judgments and scale interpretations when aggregating ordinal expert ratings. Given their superior performance under realistic patterns of expert disagreement and their robustness when such disagreement is absent, cross-national data producers are advised to adopt IRT techniques for aggregating expert-coded measures of latent concepts.
