This study evaluates 195 pre-analysis plans (PAPs) registered between 2011 and 2016 on Evidence in Governance and Politics (EGAP) and American Economic Association platforms. The research examines whether these early PAPs were sufficiently clear, precise, and comprehensive to effectively prevent post-hoc hypothesis fishing or data dredging.
Focusing on ninety-three specific projects that produced publicly available papers, the analysis assesses how faithfully their findings adhered to pre-registered specifications.
The results reveal substantial variation in both plan comprehensiveness and adherence over time. The authors discuss these mixed outcomes while weighing the benefits against potential drawbacks of PAPs as a credibility tool:
• Early PAPs often lacked sufficient detail or precision
• Many researchers failed to follow through with post-registration reporting
• Implementation challenges limited their effectiveness in addressing concerns about research integrity
The study concludes by emphasizing the need for stronger norms and institutional incentives to encourage rigorous pre-analysis planning practices.
| Pre-Analysis Plans: An Early Stocktaking was authored by George K. Ofosu and Daniel N. Posner. It was published by Cambridge in POP in 2023. |
