Why Clustered SEs Underestimate Uncertainty — A Better Method

Clustered standard errors are widely used when data include multiple observations per unit (for example, cities, states, or countries). There is active debate about the best way to estimate standard errors and confidence intervals under clustering (Harden 2011; Imbens and Kolesár 2016; MacKinnon and Webb 2017; Esarey and Menger 2019).

🔍 How the comparison was run

Extensive simulations from the literature and new simulation experiments here compared three approaches: conventional cluster-robust standard errors (CRSE), bootstrapping, and a new variance-estimation method developed in this work.

• Simulations varied the number of clusters and the amount of within-cluster variation in explanatory variables.
• Tests examined coefficient standard errors and confidence intervals for individual coefficients and sets of coefficients.

📈 Key findings

• Conventional CRSEs can seriously underestimate coefficient standard errors and associated confidence intervals, especially when the number of clusters is small and when explanatory variables show little variation within clusters.
• The new method developed here yields substantially more reliable estimates of coefficient standard errors and confidence intervals than CRSE.
• In extreme conditions the new method can still slightly underestimate confidence intervals for tests of individual and sets of coefficients, but the degree of underestimation is much smaller than for CRSE.
• The new method also produces more accurate standard error and confidence interval estimates than bootstrapping, which is often recommended as an alternative to CRSE.

❗ Why this matters

Underestimated standard errors lead to overconfident inference and inflated type I error rates in clustered-data settings. The evidence here points to a practical, better-performing alternative to CRSE and bootstrapping for researchers working with clustered observational data.