Detecting Nasty Data With Simple Plots of Complex Models
Abstract
A recent research report (Brandt, 2011) suggests that sexist ideologies predict increases in gender inequality at the country level. Using advanced multilevel modeling methods, the author combined aggregated individual-level data on sexism (N > 80,000) from the World Values Survey (World Values Survey Association, 2009) with country-level data on gender inequality (N < 60) from United Nations Human Development Reports. We were curious and plotted the sexism effect based on the data printed in the article. Figure 1 reveals that the effect was driven by only a few influential cases. When we replicated Brandt’s (2011) analysis in Mplus 6.1 (Muthén & Muthén, 2010), the effect dropped to nonsignificance after we excluded Switzerland, and was reduced to virtually zero after we excluded three further outliers. Our reanalysis casts serious doubts on Brandt’s (2011) conclusion. We are not suggesting that it would have been better for these influential cases to have been deleted. However, they should have been discussed so that readers could assess the robustness of the effect. Furthermore, unusual cases often tell their own important stories, which may critically inform future research (McClelland, 2002).
- DOI
- 10.1177/0956797611434541
- Sources
- openalex