Hidden Control Variables Give Many Papers Their Statistical Significance

🔎 The Question Investigated: How often do published articles depend on suppression effects to produce statistically significant results, and how often is that reliance disclosed?

🔍 What Is a Suppression Effect: Suppression effects are control-variable–induced increases in estimated effect sizes. Such effects demand scrutiny because researchers and readers expect a clear justification—especially when the statistical significance of a key finding depends on the inclusion of those controls.

📊 Reanalysis of Observational Studies: A reanalysis was conducted on observational studies published in a leading journal to assess the prevalence and disclosure of suppression effects. The analysis focused on whether adding control variables increased key effect estimates and whether articles acknowledged or justified those increases.

✅ Key Findings:

• Over 30% of articles depend on suppression effects for statistical significance.
• Although increases in key effect estimates from including control variables can be justifiable, none of the reviewed articles provided a justification or disclosed that their significance depended on suppression effects.
• The absence of disclosure persisted despite the fact that suppression effects typically trigger scrutiny from researchers seeking substantive explanations for control-induced changes.

⚠️ Why It Matters: These patterns point to a potential gap in the peer-review and editorial process: journals appear to be accepting articles whose core significance hinges on control-variable effects without readers, reviewers, or editors being made aware. This raises concerns about transparency, interpretability, and the standards applied during review of observational research.