In Brodeur, Cook, and Heyes (2020) we present evidence that instrumental variable (and to a lesser extent difference-in-difference) articles are more p-hacked than randomized controlled trial and regression discontinuity design articles. We also find no evidence that (i) articles published in the top five journals are different; (ii) the “revise and resubmit” process mitigates the problem; (iii) things are improving through time. Kranz and Pütz (2022) apply a novel adjustment to address rounding errors. They successfully replicate our results with the exception of our shakiest finding: after adjusting for rounding errors, bunching of test statistics for difference-in-difference articles is now smaller around the 5 percent level (and coincidentally larger at the 10 percent level).

The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time.

Online experiments have surged in popularity, but how credible are their published results? We examine the universe of hypotheses tested on Amazon Mechanical Turk and reported in leading journals between 2010 and 2021. We find evidence of widespread p-hacking, publication bias, and under-powered studies. The extent of the problems varies substantially across fields, with marketing particularly badly afflicted and little evidence of distortion in economics. We explore correlates of increased credibility and compare the online results to laboratory results from the same outlets, finding similar distortions in both, with bunching just above the significance threshold more pronounced on MTurk.