← Search

American Economic Review Vol. 113 No. 11 2023

Unpacking p-Hacking and Publication Bias

Abel Brodeur1; Scott Carrell2; David Figlio3; Lester Lusher4

1 Department of Economics, University of Ottawa and Institute for Replication (email: ) · 2 Department of Economics, University of Texas at Austin, NBER, and IZA (email: ) · 3 Department of Economics and Warner School of Education, University of Rochester, NBER, and IZA (email: ) · 4 Department of Economics, University of Pittsburgh and IZA (email: )

Abstract

We use unique data from journal submissions to identify and unpack publication bias and p-hacking. We find initial submissions display significant bunching, suggesting the distribution among published statistics cannot be fully attributed to a publication bias in peer review. Desk-rejected manuscripts display greater heaping than those sent for review; i.e., marginally significant results are more likely to be desk rejected. Reviewer recommendations, in contrast, are positively associated with statistical significance. Overall, the peer review process has little effect on the distribution of test statistics. Lastly, we track rejected papers and present evidence that the prevalence of publication biases is perhaps not as prominent as feared.

DOI
10.1257/aer.20210795
Volume
113
Issue
11
Pages
2974-3002
Language
en
Sources
bibtex:phds-export.bib openalex crossref

Cite