While Kosowski et al. (2006, Journal of Finance 61, 2551–2595) and Fama and French (2010, Journal of Finance 65, 1915–1947) both evaluate whether mutual funds outperform, their conclusions are very different. We reconcile their findings. We show that the Fama‐French method suffers from an undersampling problem that leads to a failure to reject the null hypothesis of zero alpha, even when some funds generate economically large risk‐adjusted returns. In contrast, Kosowski et al. substantially overreject the null hypothesis, even when all funds have a zero alpha. We present a novel bootstrapping approach that should be useful to future researchers choosing between the two approaches.

Multiple testing plagues many important questions in finance such as fund and factor selection. We propose a new way to calibrate both Type I and Type II errors. Next, using a double‐bootstrap method, we establish a t ‐statistic hurdle that is associated with a specific false discovery rate (e.g., 5%). We also establish a hurdle that is associated with a certain acceptable ratio of misses to false discoveries (Type II error scaled by Type I error), which effectively allows for differential costs of the two types of mistakes. Evaluating current methods, we find that they lack power to detect outperforming managers.