Expert Disagreement: Mean vs Median Responses Lead to Different Survey Outcomes

Expert Surveys & Measurement Challenges

Political scientists frequently employ expert surveys to uncover the intrinsic traits of political entities. However, experts often disagree due to varying levels of information and analytical capability.

Proposed Solutions for Better Assessment

This article addresses how to effectively handle this uncertainty:

• Assessing experts' predictive abilities (how well they judge latent positions)
• Developing robust methods to combine their diverse responses

Case Study: EU Integration & Party Politics Survey

The authors analyze prominent expert survey data from the literature focusing on party politics and European Union integration. This real-world application highlights practical consequences of different aggregation approaches.

Monte Carlo Simulation Findings

Using simulation, we show that:

• Aggregating responses using median or modal values outperforms mean averaging in most scenarios

These methods produce more accurate aggregated estimates despite expert disagreement.