Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.

Lifestyle modifications for non-alcohol related fatty liver disease: a network meta-analysis.

Al-Muzafar 2017Al-Muzafar HM, Amin KA. Probiotic mixture improves fatty liver disease by virtue of its action on lipid profiles, leptin, and inflammatory biomarkers.

Mental health difficulties are prevalent in autistic people with ~14%–50% having experienced depression and ~40%–80% having experienced anxiety disorders. Identifying interventions that improve autistic people’s mental health is a top priority. However, at present, there is no high-quality network meta-analysis of benefits and harms of different interventions. We conducted a systematic review and network meta-analysis of randomised controlled trials, searching MEDLINE, EMBASE, other databases, and trial registers until 17 October 2020. We included randomised controlled trials reporting anxiety or depression in a suitable format. We calculated effect estimates and 95% credible intervals using Bayesian network meta-analysis. Our search identified 13,794 reports, of which 71 randomised controlled trials (3630 participants) were eligible for inclusion. All trials had high risk of bias. The follow-up period ranged from 1 to 24 months. Evidence indicates uncertainty about the effects of different interventions, with more high-quality evidence needed. Available evidence suggests that some forms of cognitive behavioural therapy may decrease anxiety and depression scores in autistic children and adults; mindfulness therapy may decrease anxiety and depression scores in autistic adults with previous mental health conditions; and behavioural interventions may provide some benefit for depression in autistic children. We recommend that autistic people are given access to mental health interventions available to non-autistic people, following principles of person-centred care. PROSPERO registration ID: CRD42019136093 Lay Abstract Nearly three out of four autistic people experience mental health problems such as stress, anxiety or depression. The research already done does not guide us on how best to prevent or treat mental health problems for autistic people. Our aim was to look at the benefits and harms of different interventions on mental health outcomes in autistic people. We searched all the published randomised controlled trials (RCTs) about interventions for mental health conditions in autistic people until 17 October 2020. We also searched for RCTs that were not published in peer-reviewed journals. These were obtained from registers of clinical trials online. We then combined the information from all these trials using advanced statistical methods to analyse how good the interventions are. Seventy-one studies (3630 participants) provided information for this research. The studies reported how participants were responding to the intervention for only a short period of time. The trials did not report which interventions worked for people with intellectual disability. In people without intellectual disability, some forms of cognitive behavioural therapy and mindfulness therapy may be helpful. However, further research is necessary. Many trials used medications to target core features of autism rather than targeting mental health conditions, but these medications did not help autistic people. Until we have more evidence, treatment of mental health conditions in autistic people should follow the evidence available for non-autistic people. We plan to widely disseminate the findings to healthcare professionals through medical journals and conferences and contact other groups representing autistic people.

Meta-analyses typically quantify heterogeneity of results, thus providing information about the consistency of the investigated effect across studies. Numerous heterogeneity estimators have been devised. Past evaluations of their performance typically presumed lack of bias in the set of studies being meta-analysed, which is often unrealistic. The present study used computer simulations to evaluate five heterogeneity estimators under a range of research conditions broadly representative of meta-analyses in psychology, with the aim to assess the impact of biases in sets of primary studies on estimates of both mean effect size and heterogeneity in meta-analyses of continuous outcome measures. To this end, six orthogonal design factors were manipulated: Strength of publication bias; 1-tailed vs. 2-tailed publication bias; prevalence of p-hacking; true heterogeneity of the effect studied; true average size of the studied effect; and number of studies per meta-analysis. Our results showed that biases in sets of primary studies caused much greater problems for the estimation of effect size than for the estimation of heterogeneity. For the latter, estimation bias remained small or moderate under most circumstances. Effect size estimations remained virtually unaffected by the choice of heterogeneity estimator. For heterogeneity estimates, however, relevant differences emerged. For unbiased primary studies, the REML estimator and (to a lesser extent) the Paule-Mandel performed well in terms of bias and variance. In biased sets of primary studies however, the Paule-Mandel estimator performed poorly, whereas the DerSimonian-Laird estimator and (to a slightly lesser extent) the REML estimator performed well. The complexity of results notwithstanding, we suggest that the REML estimator remains a good choice for meta-analyses of continuous outcome measures across varied circumstances.