American Economic Review Vol. 110 No. 11 2020

Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics

Abel Brodeur¹; Nikolai Cook¹; Anthony Heyes²

¹ Department of Economics, University of Ottawa (email: ) · ² Department of Economics, University of Ottawa and University of Exeter Business School (email: )

Abstract

The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-in-differences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals, we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that (i) papers published in the Top 5 journals are different to others; (ii) the journal “revise and resubmit” process mitigates the problem; (iii) things are improving through time.

DOI

10.1257/aer.20190687

Volume

110

Issue

11

Pages

3634-3660

Language

en

Sources

bibtex:phds-export.bib openalex crossref

Cite

APA 7 MLA 9 Chicago (author–date) Harvard Elsevier Harvard (JFE, JBF…) The Accounting Review Academy of Management American Marketing Association INFORMS (Management Science) Copy citation Copied ✓