Fast principal component analysis of large data sets based on information extraction

Mizaikoff

2003

Self Cite

Many spectroscopic monitoring techniques make use of chemometric algorithms such as principal component regression (PCR) for calibration and evaluation of optical spectra. However, standard techniques of chemometric calibration and evaluation suffer from uncalibrated spectral features which are contained in measured spectra but were not considered during the initial calibration. For two main reasons such uncalibrated spectral features have to be detected and corrected: (a) if unknown spectral features appear after calibration major errors of concentration prediction are encountered and (b) the detection of uncalibrated spectral features enables recognition and handling of failures occurring in chemical processes. Hence detection, correction and characterization of uncalibrated absorption features are of great importance in on-line spectroscopy. A novel approach is proposed here for this purpose. This algorithm performs a qualitative data analysis for detecting and correcting such uncalibrated spectral features. Following this qualitative data analysis and if necessary correction, a conventional PCR is used for calculating the concentration of the calibrated analytes. Both steps together set up a so-called secured PCR (sPCR). In the undisturbed case this sPCR algorithm is demonstrated to provide equivalent results as conventional PCR and to be superior in evaluating disturbed spectra. For testing purposes simulated spectra sets are used since the disturbance is known and the determined results can be validated with respect to the magnitude of the true disturbance. The errors averaged over 1000 disturbed test samples could be decreased up to 71% by sPCR compared with conventional PCR. In up to 94% of the test samples more accurate concentration values were obtained. The novel sPCR algorithm was also applied to experimental spectra measured with mid-infrared attenuated total reflection spectroscopy considering chlorinated hydrocarbons dissolved in water. sPCR is capable to extract uncalibrated features from these experimental data and to improve the concentration results.

Section: Introductionmentioning

confidence: 99%

Secured PCR (sPCR) for detection and correction of PCR calibration model failures induced by uncalibrated spectral features

Mizaikoff

2003

Self Cite

“…For the lowest compression, however, acceleration factors are in the range <1-2, that is in two cases the compression-based approach was slightly slower the conventional PARAFAC. This is the first case in several applications of wavelet compression [29][30][31]37,38] where not enough data reduction was achieved to over-compensate computation efforts for the compression.…”

Section: Resultsmentioning

confidence: 98%

“…For example, 1D WTs have been utilized prior to principal component analysis/regression (PCA/PCR) in order to accelerate calculations [29][30][31]. Although (inverse) WTs add computational expense, which is linear with respect to the amount of data [16], smaller data sets allow for decreasing PCA computation times in the second and third order [32].…”

Section: Introductionmentioning

confidence: 99%

Accelerating the analyses of 3‐way and 4‐way PARAFAC models utilizing multi‐dimensional wavelet compression

Cramer

Kim

et al. 2005

Parallel factor analysis (PARAFAC) is one of the most popular methods for evaluating multi-way data sets, such as those typically acquired by hyphenated measurement techniques. One of the reasons for PARAFAC popularity is the ability to extract directly interpretable chemometric models with little a priori information and the capability to handle unknown interferents and missing values. However, PARAFAC requires long computation times that often prohibit sufficiently fast analyses for applications such as online sensing. An additional challenge faced by PARAFAC users is the handling and storage of very large, high-dimensional data sets. Accelerating computations and reducing storage requirements in multi-way analyses are the topics of this manuscript. This study introduces a data pre-processing method based on multi-dimensional wavelet transforms (WTs), which enables highly efficient data compression applied prior to data evaluation. Because multidimensional WTs are linear, the intrinsic underlying linear data construction is preserved in the wavelet domain. In almost all studied examples, computation times for analyzing the much smaller, compressed data sets could be reduced so much that the additional effort for wavelet compression was more than recompensated. For 3-way and 4-way synthetic and experimental data sets, acceleration factors up to 50 have been achieved; these data sets could be compressed down to a few per cent of the original size. Despite the high compression, accurate and interpretable models were derived, which are in good agreement with conventionally determined PARAFAC models. This study also found that the wavelet type used for compression is an important factor determining acceleration factors, data compression ratios and model quality.

“…individually, thus independently optimizing spectral data evaluation. For the first estimate of 2 the proposed selection algorithm is based on considering the sums of squares (SS) [19] of the reduced model (3) and the full model (4). SS is determined from that part of m meas which can be acquired by the PCs:…”

Section: Theorymentioning

confidence: 99%

“…Usually, calibration spectra are written in the rows of a calibration matrix M cal . However, as shown by Vogt and Tacke [3,4], it is advantageous to write calibration spectra in the columns of M cal if N > K, since the computational effort can be considerably decreased. Furthermore, considering PCs as columns of an overdetermined equation system is more intuitive, following the usual form applied in multivariate regression analysis.…”

Section: Introductionmentioning

confidence: 98%

Dynamic determination of the dimension of PCA calibration models using F‐statistics

Mizaikoff

2003

Self Cite

Owing to experimental measurement errors, determination of the proper dimension of calibration models is difficult. Cross-validation is a common approach for this purpose; however, if data evaluation is based on PCA only without consideration of sample concentrations, this computationally expensive method cannot be applied. In this study a statistical method for determining the proper dimension of PCA calibration models is presented from the viewpoint of multivariate regression analysis considering only measured data. For this iterative algorithm, individual principal components are included stepwise in a reduced model, which is subsequently tested against the full model including all principal components. This algorithm can be individually applied for optimized data evaluation to every measured data vector such as an optical spectrum of chemical analyte. This comparison is performed by an F-test comparing estimates of residual variance of a measurement spectrum determined from the reduced and the full model. This approach determines a lack of fit due to insufficient principal components. If no lack of fit is evident for a certain reduced model, it is considered that a sufficiently large model has been found and inclusion of additional principal components is stopped. Hence the resulting reduced calibration model includes only statistically significant principal components (PCs) and determines the minimum number of required PCs for a given measurement spectrum. The proposed algorithm is initially investigated using simulated data and subsequently applied to three different experimental sets of spectra. It is shown that for synthetic data at reasonable noise levels the correct number of principal components can be determined in most cases. The experimental examples demonstrate that the number of principal components determined by the proposed algorithm is slightly larger than a user would select manually by subjective visual inspection. As one result, the algorithm is able to detect small but significant spectroscopic features of experimental data which would otherwise be neglected.