Quality assessment of meta-analyses on soil organic carbon

Abstract

Abstract. Soil organic carbon (SOC) plays a vital role in the global carbon cycle and is a potential sink for carbon dioxide. Agricultural management practices can support carbon sequestration and, therefore, offer potential removal strategies whilst also improving overall soil quality. Meta-analysis allows one to summarize results from primary articles by calculating an overall effect size and to reveal the source of variation across studies. The number of meta-analyses published in the field of agriculture is continuously rising. At the same time, more and more articles refer to their synthesis work as a meta-analysis, despite applying less than rigorous methodologies. As a result, poor-quality meta-analyses are published and may lead to questionable conclusions and recommendations to scientists, policymakers, and farmers. This study aims at quantitatively analyzing 31 meta-analyses, published between the years of 2005 and 2020, studying the effects of different management practices on SOC. We compiled a set of quality criteria suitable for soil and agricultural sciences by adapting existing meta-analytical guidelines from other disciplines. The set is supported by a scoring scheme that allows for a quantitative analysis. The retrieved meta-analyses were structured according to 11 management categories, such as tillage, cover crops, crop residue management, and biochar application, which allowed us to assess the state of knowledge on these categories. Major deficiencies were found in the use of standard metrics for effect size calculation, independence of effect sizes, standard deviation extraction for each study, and study weighting by the inverse of variance. Only 1 out of 31 SOC meta-analyses, which studied the effects of no tillage/reduced tillage compared with conventional tillage, was found to be of high quality. Therefore, improved meta-analyses on the effects of organic agriculture, biochar, fertilization, or crop diversification on SOC are urgently needed. We conclude that, despite efforts over the last 15 years, the quality of meta-analyses on SOC research is still low. Thus, in order for the scientific community to provide high-quality synthesis work and to make advancements in the sustainable management of agricultural soils, we need to adapt rigorous methodologies of meta-analysis as quickly as possible.

Cover crops affect pool specific soil organic carbon in cropland – A meta‐analysis

“…Often, as each additional fraction analysed increases the workload significantly (Poeplau et al, 2018), measurement of MAOC is neglected. Researchers also tend to calculate MAOC by subtracting POC from total SOC, thereby estimating parameters that cannot be used in high‐quality meta‐analysis, which only include measured response variables (Fohrafellner, Zechmeister‐Boltenstern, Murugan, & Valkama, 2023). We therefore encourage scientists to assess the stable fraction analytically in their experiments, as there are still uncertainties on how CCs and related moderators impact MAOC.…”

“…Our meta‐analysis synthesized global data on CC effects on SOC pools, focusing on agricultural management and climatic zones which are also found in Europe. At the same time, we followed strict quality criteria for conducting agricultural meta‐analyses (Borenstein et al, 2009; Fohrafellner, Zechmeister‐Boltenstern, Murugan, & Valkama, 2023; Koricheva et al, 2013). Therefore, our meta‐analysis provides novel information and high‐quality data on the responses of SOC pools to CCs, cultivated under conditions representative of Europe.…”

“…There, we further describe that, as available data on the MAOC and POC pool was scarce, we also included organic matter data in our analysis, namely MAOM and POM. This was possible, as the effect size of choice (log response ratio) allows to summarize values with a large variation across studies (Fohrafellner, Zechmeister‐Boltenstern, Murugan, & Valkama, 2023). Therefore, effect sizes, calculated from organic matter or organic carbon, can be compared with each other.…”

“…Consequently, it can significantly contribute to enhancing soil health, a matter of paramount concern in Europe's mission “A Soil Deal for Europe” (Commission et al, 2021). We therefore conducted the first global meta‐analysis, investigating the effects of CCs on the mineral‐associated organic carbon (MAOC), particulate organic carbon (POC) and MBC pool, that only included experimental studies conducted in climate zones that are relevant to Europe, whilst also following strict quality criteria (Borenstein et al, 2009; Fohrafellner, Zechmeister‐Boltenstern, Murugan, & Valkama, 2023; Koricheva et al, 2013). This approach was chosen as not enough studies based on European experiments were available to conduct a meta‐analysis.…”

Collected knowledge on the impacts of agricultural soil management practices in Europe

“…Research on sustainable soil management of agricultural soils is reflected in a wide body of literature. Some examples that can be pointed to are the prevention of soil erosion (Du et al, 2022;Ebabu et al, 2022), the decline of organic matter (Fohrafellner et al, 2023;Gross & Glaser, 2021;Martínez-García et al, 2018), and prevention of soil salinisation (Cuevas et al, 2019;Daliakopoulos et al, 2016;Paz, Amezketa, et al, 2023), just to mention three soil threats. However, despite its vital importance, soil is facing degradation globally (FAO, 2015).…”