Calculating the probability of random sampling for continuous variables in submitted or published randomised controlled trials

Abstract

SummaryIn a previous paper, one of the authors (JBC) used a chi-squared method to analyse the means (SD) of baseline variables, such as height or weight, from randomised controlled trials by Fujii et al., concluding that the probabilities that the reported distributions arose by chance were infinitesimally small. Subsequent testing of that chi-squared method, using simulation, suggested that the method was incorrect. This paper corrects the chi-squared method and tests its performance and the performance of Monte Carlo simulations and ANOVA to analyse the probability of random sampling. The corrected chi-squared method and ANOVA method became inaccurate when applied to means that were reported imprecisely. Monte Carlo simulations confirmed that baseline data from 158 randomised controlled trials by Fujii et al. were different to those from 329 trials published by other authors and that the distribution of Fujii et al.'s data were different to the expected distribution, both p < 10 À16 . The number of Fujii randomised controlled trials with unlikely distributions was less with Monte Carlo simulation than with the 2012 chi-squared method: 102 vs 117 trials with p < 0.05; 60 vs 86 for p < 0.01; 30 vs 56 for p < 0.001; and 12 vs 24 for p < 0.00001, respectively. The Monte Carlo analysis nevertheless confirmed the original conclusion that the distribution of the data presented by Fujii et al. was extremely unlikely to have arisen from observed data. The Monte Carlo analysis may be an appropriate screening tool to check for non-random (i.e. unreliable) data in randomised controlled trials submitted to journals.

Data fabrication and other reasons for non‐random sampling in 5087 randomised, controlled trials in anaesthetic and general medical journals

“…Techniques have been developed to analyse baseline variables, particularly the mean (SD) of continuous variables, and these have helped to identify fabricated data in randomised, controlled trials by Fujii et al [1]. The general principles of these methods have been explained elsewhere [2,3]. The same approach has been used to investigate trials published by Yuhji Saitoh, a co-author of Dr Fujii, and form a component of the investigation of his work [4].…”

“…The purpose of this survey is to assess if: (1) the distribution of baseline means corresponded to the expected distribution and whether discrepancies were shared by leading non-anaesthetic vs. anaesthetic journals; (2) there was a different rate of retraction in leading non-anaesthetic vs. anaesthetic journals; and (3) data corruption was discoverable by the new statistical techniques in those papers/authors that had been retracted. I used the method to detect anomalies in the distributions of baseline variable mean (SD) from randomised, controlled trials published during 15 years in six specialist anaesthetic journals (Anaesthesia, Anesthesia and Analgesia, Anesthesiology, the British Journal of Anaesthesia, the Canadian Journal of Anesthesia and the European Journal of Anaesthesiology) and two general medical journals (Journal of the American Medical Association (JAMA) and New England Journal of Medicine (NEJM)).…”

What Anaesthesia is doing to combat scientific misconduct and investigate data fabrication and falsification

“…The first publication of the use of the Carlisle Method was the investigation into randomised controlled trials published by Yoshitaka Fujii, which led to 138 retractions (to date) and Fujii's legacy [5]. Dr John Carlisle then published an improvement of his methodology using Monte Carlo simulation [6]. Dr Yuhji Saitoh was a co-author of Dr Fujii in 32 of the retracted papers.…”

“…© 2016 The Association of Anaesthetists of Great Britain and Ireland Anaesthesia 2017, 72,[3][4][5][6][7][8][9][10][11][12][13][14][15][16] Editorial…”

Deaths by horsekick in the Prussian army - and other ‘Never Events’ in large organisations

“…statistically improbable) with this assumption. A concrete example of this distinction is the recent analysis by Carlisle, who concluded that the data of Fujii were very unlikely to have been derived from random sampling, and were therefore more likely deterministically derived [9][10][11][12].…”