Expression of Concern: Effort for Payment: A Tale of Two Markets
Abstract
This statement is an Expression of Concern regarding the article “Effort for Payment: A Tale of Two Markets” (Heyman & Ariely, 2004) published in Psychological Science. This Expression of Concern is prompted by some uncertainty regarding the values of statistical tests reported in the article and the analytic approach taken to the data. The corresponding author of the article and coauthor of this statement, Dan Ariely, attempted to locate the original data in an effort to resolve the ambiguities but was unsuccessful. Because the ambiguities cannot be resolved, we decided to issue an Expression of Concern about the confidence that can be held in the results reported in the article. The ambiguity originally was brought to the attention of the Editor in Chief by Gilad Feldman, Hirotaka Imada, Wan Fei Chan, Yuk Ki Ng, Lee Hing Man, Mei Sze Wong, and Bo Ley Cheng. These researchers ran the article through statcheck (Epskamp & Nuijten, 2018), an R package that is designed to detect statistical errors (much like spell check and grammar check in Word). The program searches articles for statistical results, recalculates the values, and compares the reported and recalculated values to determine whether they match. Statcheck is now required for all articles published in Psychological Science. However, it was not available at the time of submission and acceptance of Heyman and Ariely’s article. The statcheck analysis conducted by Feldman and colleagues produced some discrepancies between the statistical values reported in the article and those determined by the recalculation. Gilad Feldman notified the editor of these discrepancies. On receipt of Feldman and colleagues’ report, we reran the statcheck analysis in house. The report yielded 13 “errors” or mismatches. Five of the discrepancies were associated with Experiment 1 of the article, four with Experiment 2, and four with Experiment 3. The values reported in the article and those resulting from the statcheck recalculation are provided in Table 1. In seven of the instances of discrepancy between the reported and recalculated values, the p value of the test changed, but the status of the test—as statistically significant or not statistically significant—remained the same. That is, the test was reported as statistically significant and remained so or was reported as nonsignificant and remained so. Thus, these discrepancies do not impact the interpretation or conclusions. In six of the instances of discrepancy, the outcome of the statistical test was different, resulting in what is known as a “decision error.” In these instances, there is a mismatch between results reported as significant and the results of the recalculations, which indicate that the tests are not statistically significant. These discrepancies, if the recalculated values are correct, would change the interpretation of the data in a manner that substantively alters the conclusions drawn from the research. In addition to the issues raised by the statcheck report, there is a lack of specificity regarding the analytic approach adopted in the article. The authors did not provide a narrative description of the analytic approach, which featured seven conditions in Experiment 1 (N = 614 participants), five conditions in Experiment 2 (N = 159), and five conditions in Experiment 3 (N = 90). In all cases, the statistical tests were F tests; for all reported tests, the degrees of freedom are indicated as (1, 607), (1, 154), and (1, 84), for Experiments 1, 2, and 3, respectively. Although the original analyses cannot be located, the authors recall conducting the analyses using a planned contrasts approach in the SuperANOVA package, using all the data from all the conditions for the error term and the specific cells for the conditions. Note that this is a unique analysis approach, and the differences from the statcheck results could be due to the way this approach was carried out or reported. Because the data for this article cannot be located, it cannot be determined whether the original reported values or those resulting from the statcheck recalculation are correct. Nor is it possible to more fully disambiguate the analytic approach. Given the ambiguities, the confidence we place in the conclusions drawn from the research is diminished. However, again, given the ambiguities, the Editor in Chief decided not to change the official publication record of the article through a Corrigendum. Instead, the corresponding author and editor are issuing this Expression of Concern and note that the differences between the values reported in the published article and the values recalculated through 1035782 PSSXXX10.1177/09567976211035782Bauer, ArielyExpression of Concern: Effort for Payment: A Tale of Two Markets research-article2021
- DOI
- 10.1177/09567976211035782
- Sources
- semanticscholar openalex