Statistical infarction: A postmortem of the Cornell Food and Brand Lab pizza publications

Abstract

We previously reported over 150 inconsistencies in a series of four articles (the "pizza papers") from the Cornell Food and Brand Lab that described a study of eating habits at an all-you-can-eat pizza buffet. The lab's initial response led us to investigate more of their work, and our investigation has now identified issues with at least 45 publications from this lab. Perhaps because of the growing media attention, Cornell and the lab have released a statement concerning the pizza papers, which included a response to the inconsistencies, along with data and code. Many of the inconsistencies were identified with the new technique of granularity testing, and this case has the highest density of granularity inconsistencies that we know of. This is also the first time a data set has been made public after granularity concerns were raised, making it a highly suitable case study for showing the accuracy and potential of this technique. It is also important that a third party audit the lab's response, given the continuing investigation of misconduct and presumably future reports and data releases. Our careful inspection of the data set suggests no evidence of fabrication, but we found the lab's report confusing, incomplete, and error prone. In addition, we found the number of missing, unusual, and logically impossible responses in the data set highly concerning. Unfortunately, given the unsound theory, poor methodology, questionable data, and countless errors, we find it remarkable that these four papers were published and recommend retraction of all four papers.

How to Be a Statistical Detective

“…The GRIM test was instrumental in uncovering a slew of numerical inconsistencies in a number of papers by the same author; these papers were later retracted …”

Ciencia abierta: iniciativas para mejorar la investigación en Latinoamérica

“…Otro caso fue el del investigador en conducta alimentaria Brian Wansink, quien en su blog comentó sobre su forma de buscar resultados en datos de un estudio fallido y de ello resultaron cuatro publicaciones. Esto llamo la atención de metacientíficos, quienes encontraron en tales artículos un gran número de inconsistencias y errores (Anaya et al, 2017). Escrutinio adicional a la investigación del profesor Wansink ha llevado a retracción de al menos dieciocho de sus publicaciones y al dictamen de incurrir en mala conducta académica (Munafò et al, 2018).…”

Recovering data from summary statistics: Sample Parameter Reconstruction via Iterative TEchniques (SPRITE)

“…SPRITE complements the existing tests known as GRIM (Brown and Heathers, 2017) and GRIMMER (Anaya, 2016), which are simple methods for detecting certain inaccuracies in published means (GRIM) and SDs (GRIMMER) using the granularity of their constituent values. Recently, both tests were central to the identification of problems in a series of papers in food psychology (van der Zee et al, 2017;Anaya et al, 2017). While they are theoretically instructive, in practice both GRIM and GRIMMER can only be applied to a restricted subset of published work since they typically require the per-cell sample size to be smaller than N=100, assuming that the descriptive statistics are reported to the usual two decimal places (dp).…”