Diffuse reflectance infrared spectroscopy estimates for soil properties using multiple partitions: Effects of the range of contents, sample size, and algorithms

Abstract

The RMSE of validation (RMSE V ) and ratio of the interquartile range to RMSE V (RPIQ V ) are key quality parameters in diffuse reflectance infrared (IR) spectroscopy studies, but the effects of different factors on these parameters are often not sufficiently considered. Our objectives were to reveal the effects of range of contents, sample size, data pretreatment, wavenumber region selection, and algorithms on the evaluation of IR spectra in the wavenumber range from 1,000 to 7,000 cm −1 (mid-and long-wave near IR) estimations. Contents of soil organic C (SOC), N, clay, and sand and pH values were determined for surface soils of an arable field in India, and IR spectra were recorded for four samples consisting of 71-263 soils. For each of the four samples, five random partitions into calibration and validation datasets were carried out, and partial least squares regression (PLSR) or support vector machine regression was performed. A plot of the RMSE V values against the interquartile ranges of measured values for the validation samples (IQR V ) indicated that the IQR V was a key parameter for all soil properties: a sufficiently high IQR V -which is affected by sample size and random partitioning-resulted in generally good estimation accuracies (RPIQ V ≥ 2.70). Optimized data pretreatment and wavenumber region selection improved estimation accuracy for SOC and pH. Support vector machine regression was superior to PLSR for the estimation of SOC, clay, and sand, but worse for pH. Overall, this study indicates that multiple partitioning of the data is essential in IR studies and suggests that RPIQ V and RMSE V need to be interpreted in the context of the respective IQR V values.

Calibration set optimization and library transfer for soil carbon estimation using soil spectroscopy—A review

“…According to the authors, the OPS-PLSR improved the prediction accuracy of SOC as compared to the full spectrum approach but only slightly (Full-spectrum model for Alfisols: R 2 = .56, RPD = 1.53 and RMSE = 0.08%; OPS-PLSR for Alfisols: 0.57, 1.54, and 0.08%). Ludwig et al (2021) investigated the effects of SOC% range, sample size, and wavenumber region selection on the RMSE and RPIQ. They used an automatic method to select optimal models from more than 17,800 combinations of nine spectral regions between 7,000 and 1,030 cm −1 (MIR and long-range NIR) and spectral preprocessing treatments.…”

“…Lower RMSEs were associated with more homogeneous calibration models and higher RMSEs with more heterogeneous models; however, as Clingensmith et al (2019) found, more heterogeneous models also had a wider IQR resulting in higher RPIQs. The authors cautioned that RPIQ and RMSE values should not be interpreted independently in infrared studies, but rather in the context of their associated IQR values (Ludwig et al, 2021).…”

“…This is an important consideration for deciding when and how to subset the calibration set. Additionally, an adequate statistical comparison of model performance across different ranges of analyte values should not only be based on RMSE, given that this will decrease as the analyte range decreases (Stenberg et al., 2010), but also requires a comparison of R 2 , RPD, and RPIQ in the context of the respective interquartile range (IQR), if possible (Ludwig et al., 2021).…”

“…Ludwig et al. (2021) investigated the effects of SOC% range, sample size, and wavenumber region selection on the RMSE and RPIQ. They used an automatic method to select optimal models from more than 17,800 combinations of nine spectral regions between 7,000 and 1,030 cm −1 (MIR and long‐range NIR) and spectral preprocessing treatments.…”

“…(2019) found, more heterogeneous models also had a wider IQR resulting in higher RPIQs. The authors cautioned that RPIQ and RMSE values should not be interpreted independently in infrared studies, but rather in the context of their associated IQR values (Ludwig et al., 2021).…”