We propose a method to test a prediction of the distribution of a stochastic process. In a non-Bayesian, non-parametric setting, a predicted distribution is tested using a realization of the stochastic process. A test associates a set of realizations for each predicted distribution, on which the prediction passes, so that if there are no type I errors, a prediction assigns probability 1 to its test set. Nevertheless, these test sets can be “small”, in the sense that “most” distributions assign it probability 0, and hence there are “few” type II errors. It is also shown that there exists such a test that cannot be manipulated, in the sense that an uninformed predictor, who is pretending to know the true distribution, is guaranteed to fail on an uncountable number of realizations, no matter what randomized prediction he employs. The notion of a small set we use is category I, described in more detail in the paper.

We consider a cross-calibration test of predictions by multiple potential experts in a stochastic environment. This test checks whether each expert is calibrated conditional on the predictions made by other experts. We show that this test is good in the sense that a true expert—one informed of the true distribution of the process—is guaranteed to pass the test no matter what the other potential experts do, and false experts will fail the test on all but a small (category I) set of true distributions. Furthermore, even when there is no true expert present, a test similar to cross-calibration cannot be simultaneously manipulated by multiple false experts, but at the cost of failing some true experts.

We study a one-sided offers bargaining game in which the buyer has private information about the value of the object and the seller has private information about his beliefs about the buyer's valuation. We show that this uncertainty about uncertainties dramatically changes the set of outcomes. In particular, second order beliefs can lead to a delay in reaching agreement even when the seller makes frequent offers. We show that not all types of second order beliefs lead to a delay. When the buyer assigns positive probability to the seller knowing the buyer's value, then delay not only can occur, but it must occur for a class of equilibria. However, in all other cases delay will never occur. Copyright The Econometric Society 2005.