We leverage a large-scale incentivized survey eliciting behaviors from (almost) an entire undergraduate university student population, a representative sample of the US population, and Amazon Mechanical Turk (MTurk) to address concerns about the external validity of experiments with student participants. Behavior in the student population offers bounds on behaviors in other populations, and correlations between behaviors are similar across samples. Furthermore, non-student samples exhibit higher levels of noise. Adding historical lab participation data, we find a small set of attributes over which lab participants differ from non-lab participants. An additional set of lab experiments shows no evidence of observer effects.

This paper studies, theoretically and empirically, the role of overconfidence in political behavior. Our model of overconfidence in beliefs predicts that overconfidence leads to ideological extremeness, increased voter turnout, and stronger partisan identification. The model also makes nuanced predictions about the patterns of ideology in society. These predictions are tested using unique data that measure the overconfidence and standard political characteristics of a nationwide sample of over 3,000 adults. Our numerous predictions find strong support in these data. In particular, we document that overconfidence is a substantively and statistically important predictor of ideological extremeness, voter turnout, and partisan identification.

We study the design of randomized controlled experiments when outcomes are significantly affected by experimental subjects' unobserved effort expenditure. While standard randomized controlled trials (RCTs) are internally consistent, the unobservability of effort compromises external validity. We approach trial design as a principal-agent problem and show that natural extensions of RCTs—which we call selective trials—can help improve external validity. In particular, selective trials can disentangle the effects of treatment, effort, and the interaction of treatment and effort. Moreover, they can help identify when treatment effects are affected by erroneous beliefs and inappropriate effort expenditure.

This paper studies the problem of experiment design by an ambiguity-averse decision-maker who trades off subjective expected performance against robust performance guarantees. This framework accounts for real-world experimenters’ preference for randomization. It also clarifies the circumstances in which randomization is optimal: when the available sample size is large and robustness is an important concern. We apply our model to shed light on the practice of rerandomization, used to improve balance across treatment and control groups. We show that rerandomization creates a trade-off between subjective performance and robust performance guarantees. However, robust performance guarantees diminish very slowly with the number of rerandomizations. This suggests that moderate levels of rerandomization usefully expand the set of acceptable compromises between subjective performance and robustness. Targeting a fixed quantile of balance is safer than targeting an absolute balance objective.