In Brodeur, Cook, and Heyes (2020) we present evidence that instrumental variable (and to a lesser extent difference-in-difference) articles are more p-hacked than randomized controlled trial and regression discontinuity design articles. We also find no evidence that (i) articles published in the top five journals are different; (ii) the “revise and resubmit” process mitigates the problem; (iii) things are improving through time. Kranz and Pütz (2022) apply a novel adjustment to address rounding errors. They successfully replicate our results with the exception of our shakiest finding: after adjusting for rounding errors, bunching of test statistics for difference-in-difference articles is now smaller around the 5 percent level (and coincidentally larger at the 10 percent level).

The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time.