← Search

Diagnosing Model Performance Under Distribution Shift

Tiffany (Tianhui) Cai¹; Hongseok Namkoong²; Steve Yadlowsky³

¹ Department of Statistics, Columbia University, New York, New York 10027 · ² Decision, Risk and Operations Division, Columbia Business School, New York, New York 10027; · ³ Google DeepMind, Mountain View, California 94043

Diagnosing Why Models Fail Under Distribution Shift and What to Do Next Predictive models often perform worse when deployed in a new target setting, but it is rarely clear why. In “Diagnosing Model Performance Under Distribution Shift,” Cai, Namkoong, and Yadlowsky introduce a diagnostic, distribution shift decomposition (DISDE), that attributes the change in performance from the training to target distributions into terms for (i) an increase in harder but previously seen inputs from training, (ii) changes in how outcomes relate to inputs, and (iii) poor performance on new input regions absent from the training data. Applications to employment prediction demonstrate how this decomposition can inform potential modeling improvements, guiding whether to use domain adaptation techniques, adjust model covariates, or collect new samples. Additionally, DISDE is used to help explain why certain domain adaptation methods fail to improve model performance for satellite image classification.

DOI

10.1287/opre.2023.0217

Volume

74 (2)

Pages

898-916

Language

en

Export

BibTeX

Sources

crossref