Pre‐whitening of data by covariance‐weighted pre‐processing

Abstract

A data pre-processing method is presented for multichannel`spectra' from process spectrophotometers and other multichannel instruments. It may be seen as a`pre-whitening' of the spectra, and serves to make the instrument`blind' to certain interferants while retaining its analyte sensitivity. Thereby the instrument selectivity may be improved already prior to multivariate calibration. The result is a reduced need for process perturbation or sample spiking just to generate calibration samples that span the unwanted interferants. The method consists of shrinking the multidimensional data space of the spectra in the off-axis dimensions corresponding to the spectra of these interferants. A`nuisance' covariance matrix S is first constructed, based on prior knowledge or estimates of the major interferants' spectra, and the scaling matrix G = S À1/2 is defined. The preprocessing then consists of multiplying each input spectrum by G. When these scaled spectra are analysed in conventional chemometrics software by PCA, PCR, PLSR, curve resolution, etc., the modelling becomes simpler, because it does not have to account for variations in the unwanted interferants. The obtained model parameter may finally be descaled by G À1 for graphical interpretation. The pre-processing method is illustrated by the use of prior spectroscopic knowledge to simplify the multivariate calibration of a fibre optical vis/NIR process analyser. The 48-dimensional spectral space, corresponding to the 48 instrument wavelength channels used, is shrunk in two of its dimensions, defined by the known spectra of two major interferants. Successful multivariate calibration could then be obtained, based on a very small calibration sample set. Then the paper shows the pre-whitening used for reducing the number of bilinear PLSR components in multivariate calibration models. Nuisance covariance S is either based on the prior knowledge of interferants' spectra or based on estimating the interferants' spectral subspace from the calibration data at hand. The relationship of the pre-processing to weighted and generalized least squares from classical statistics is outlined.

Quantitative Big Data: where chemometrics can contribute

“…These may be implemented in different ways, e.g. by Generalized Least Squares preprocessing (matrix multiplication by the square root of expected covariances [30]), or by statistically more advanced Bayesian analyses. Unfortunately, Evolution did not allow our brain to do matrix multiplication-otherwise we humans might have been able to use this to correct for our own cultural biases.…”

Framework for explicit drift correction in multivariate calibration models

“…For example, if the drift is a sample-invariant constant baseline, this drift is not captured inD since Equation (14) and thus Assumption A2 do not hold. In References [15,16,18,29], q correction samples are used for calibration transfer between t = 2 instruments. Here, a reasonable choice of X m corresponds to the spectroscopic measurements on the instrument corresponding to the calibrated instrument (say, first instrument).…”

“…In GLS [15,16],b i is computed from the calibration data pair {X c W, y c,i }, where the [n w × n w ] weighting matrix W is used to downweigh (or shrink) the drift subspace:…”

“…component correction (CC) [13], independent interference reduction (IIR) [14], generalized least squares (GLS) [15,16], external parameter orthogonalization (EPO) [17], calibration transfer by orthogonal projection (TOP) [18], dynamic orthogonal projection (DOP) [10,19,20], difference correction of prediction spectra (DCPS) [21], error removal by orthogonal subtraction (EROS) [22], and orthogonal independent component regression (O-ICR) [29]. Confronted with a choice from this alphabet soup of ECMs only adds to the confusion of a practitioner.…”

Mapping the Spectral Soil Quality Index (SSQI) Using Airborne Imaging Spectroscopy

“…The spectral analysis procedure involves the preparation of soil samples, spectral acquisition, the preprocessing of spectral data, and the selection of an appropriate statistical model and each step can affect the accuracy of the model for each individual indicator [16]. Different pre-processing transformations (PPTs) have been applied to improve prediction ability [17,18], such as mean normalization, baseline offset, maximum normalization [17], first derivatives [19], the Savitzky-Golay smoothing algorithm [20], second derivatives [21], and generalized least squares weighting (GLSW) [22]. Such methods usually remove baseline effects and spectral noise, and reduce the impact of particle size [23].…”