Statistical Reform in Psychology
Abstract
We investigated whether statistical practices in psychology have been changing since 1998.Early in this period, the American Psychological Association (APA) Task Force on Statistical Inference (TFSI; Wilkinson & TFSI, 1999) advocated improved statistical practices, including reporting effect sizes and confidence intervals (CIs).The APA (2001) Publication Manual included expanded advice on effect sizes and stated that CIs are ''in general, the best reporting strategy' ' (p.22).Any changes since 1998 may, of course, have had many causes other than those two sources of advice. STATISTICAL PRACTICES IN LEADING JOURNALSWe examined 10 leading international psychology journals that publish mainly empirical research; Figure 1 shows the journals.For each journal, we coded 40 empirical articles published in each of three periods: the first 40 published in 1998, the most recent 40 available for 2003-2004 when we coded in mid-2004, and the most recent 40 available for 2005-2006 around April 2006.We focused on three statistical practices central to the statistical-reform debate that we could code reliably: We noted any use of null-hypothesis significance testing (NHST), CIs, and figures with error bars.In addition, we searched for any discussion of CIs or error bars, or use of these to support interpretation.(Finch et al., 2004, found that even when CIs are reported, they are rarely interpreted.)We did not code effect sizes, but have elsewhere documented increases in some effect size reporting, for example, in the Journal of Consulting and Clinical Psychology (JCCP;Fidler et al., 2005).NHSTwas used in almost all articles (97.8, 97.7, and 96.9% in the three time periods, respectively).There was a substantial increase in figures with error bars (11.0, 24.7, and 37.8%; see Fig. 1, left panel; note that the CI around each mean difference does not include the mean of the previous time period, which indicates a statistically significant change from the previous time period, p < .05,two-tailed; or p rep > .92).The error bars were sometimes CIs (12.7, 9.5, and 13.8% of articles with figures with error bars), but more often represented standard errors (58.7, 43.1, and 46.7%) or were not labeled (33.0, 28.8, and 33.8%)-a serious omission (Cumming & Finch, 2005).The Journal of Experimental Psychology: General (JEPG; 12.5, 62.5, and 75%) showed the largest increase in figures with error bars, and every journal except Child Development, the Journal of Abnormal Psychology, and JCCP showed an increase from 1998 to 2005-2006 that was larger than the lower arm of the CI around the difference between those two times (examples for some journals are shown by the light error bars in Fig. 1, left panel).CIs were rarely reported, but their use increased somewhat (3.7, 9.2, and 10.6%; Fig. 1, right panel).The mean increase from 1998 to [2003][2004] was larger than the lower arm of the CI around the difference.The Journal of Abnormal Psychology, JCCP, and JEPG showed the largest increases, and they and Child Development and the Journal of Abnormal Child Psychology showed increases larger than the lower arm of the CI around the difference between 1998 and 2005-2006.Most CIs were reported in the text or a table (78.7, 64.2, and 71.5%), rather than as error bars in figures.Only 24.1% of articles with CIs had any interpretation of a CI, and such interpretation was often in terms of NHST, rather than width or precision (Cumming & Finch, 2005).Only 3 of the 168 articles with standard error bars in a figure included any interpretation of error bars.We examined editorials, Web sites, and instructions to authors for the 10 journals.Most offered little statistical advice, and there was little change since 1997.Exceptions were the Web site for JEPG, which since June 2004 has provided extensive statistical advice (http://web.uvic.ca/jepgen/tips.htm);editorials in
- DOI
- 10.1111/j.1467-9280.2007.01881.x
- Sources
- openalex