Decision makers, such as doctors, judges, and managers, make consequential choices based on predictions of unknown outcomes. Do these decision makers make systematic prediction mistakes based on the available information? If so, in what ways are their predictions systematically biased? In this article, I characterize conditions under which systematic prediction mistakes can be identified in empirical settings such as hiring, medical diagnosis, and pretrial release. I derive a statistical test for whether the decision maker makes systematic prediction mistakes under these assumptions and provide methods for estimating the ways the decision maker’s predictions are systematically biased. I analyze the pretrial release decisions of judges in New York City, estimating that at least 20% of judges make systematic prediction mistakes about misconduct risk given defendant characteristics. Motivated by this analysis, I estimate the effects of replacing judges with algorithmic decision rules and find that replacing judges with algorithms where systematic prediction mistakes occur dominates the status quo.

A researcher can use a tightly parameterized structural model to obtain internally consistent estimates of a wide range of economically interesting targets. We ask how reliable these estimates are when the researcher’s model may be misspecified. We focus on the case of multivariate, potentially nonlinear models where the causal variable of interest is endogenous. Reliable estimates require that the researcher’s model is flexible enough to describe the effects of the endogenous variable approximately correctly. Reliable estimates do not require that the researcher has correctly specified the role of the exogenous controls in the model. However, if the role of the controls is misspecified, reliable estimates require a property we call strong exclusion. Strong exclusion depends on having sufficiently many instruments that are unrelated to the controls. We discuss how practitioners can achieve strong exclusion, and illustrate our findings with an application to a differentiated goods model of demand for beer.