The Basel III post-crisis reforms target the application of internal credit risk models for the estimation of the risk weighted assets of banks due to concerns about model risk. We use a unique dataset of 4.9 million probability of default (PD) estimates covering the January 2016 - June 2020 period sourced from 28 global banks to provide a deep insight into the comparability of model outputs. Our contribution is four-fold. Firstly, we confirm that there is a substantial variance in credit risk estimates. Secondly, we show that the level of PD variance is dependent on the entity type, industry , and location. Thirdly, we conclude that a considerable part of the variance is systematic, especially for credit risk estimates of funds. Finally, we illustrate the massive impact of the COVID-19 pandemic on the PD variance. The results highlight areas with relatively larger comparability issues, and they can be used by regulators to design more targeted policies.