Simple and Effective Way for Data Preprocessing Selection Based on Design of Experiments

Abstract

The selection of optimal preprocessing is among the main bottlenecks in chemometric data analysis. Preprocessing currently is a burden, since a multitude of different preprocessing methods is available for, e.g., baseline correction, smoothing, and alignment, but it is not clear beforehand which method(s) should be used for which data set. The process of preprocessing selection is often limited to trial-and-error and is therefore considered somewhat subjective. In this paper, we present a novel, simple, and effective approach for preprocessing selection. The defining feature of this approach is a design of experiments. On the basis of the design, model performance of a few well-chosen preprocessing methods, and combinations thereof (called strategies) is evaluated. Interpretation of the main effects and interactions subsequently enables the selection of an optimal preprocessing strategy. The presented approach is applied to eight different spectroscopic data sets, covering both calibration and classification challenges. We show that the approach is able to select a preprocessing strategy which improves model performance by at least 50% compared to the raw data; in most cases, it leads to a strategy very close to the true optimum. Our approach makes preprocessing selection fast, insightful, and objective.

Development of an on-line Raman spectral analytical method for monitoring and endpoint determination of the Cornu Caprae Hircus hydrolysis process

“…Compared with the method proposed by Gerretzen et al ,. no transformation for responses (RMSECVs) was applied instead of the logarithm in this study.…”

“…For optimal preprocessing method selection, a DOE‐based method was used based on the study conducted by Gerretzen et al . However, a different approach to simulating experimental replicates was conducted.…”

“…All preprocessing methods considered in this study are shown in Table , and the preprocessing steps were applied in the order of smoothing (A), baseline correction (B), scatter correction (C) and scaling (D). Instead of the ‘trial‐and‐error’ method, a DOE‐based approach for preprocessing step evaluation was used based on the method proposed by Gerretzen et al . These four preprocessing steps were considered as four factors, and steps that had significant effects on the responses, model performance (RMSECVs), were identified by the DOE‐based approach.…”

“…The low level (−) and the high level (+) of each factor are shown in Table , and 16 experiments were designed (Table ). The low level indicates not performing the step or is a customary way for model development. The high level is selected from the available preprocessing methods for each step, provided the selected method has the preprocessing function of the corresponding step.…”

Optimization of ion trajectories in a dynamically harmonized Fourier‐transform ion cyclotron resonance cell using a design of experiments strategy

“…Design of experiments (DoE) is a rigorous procedure in which defined modifications are induced based on a given set of variables. The response of the system is recovered to create a mathematical model that allows a prediction of a set of parameters optimizing the response value . Some examples of the optimization of mass spectrometer using DoE include the work performed by Lemonakis et al and Tebani et al Here, this method is applied on a defined response, the sum of the abundances of the second harmonic of sodium trifluoroacetate cluster.…”