Application of SIC (simple interval calculation) for object status classification and outlier detection—comparison with regression approach

Abstract

We introduce a novel approach termed simple interval calculation (SIC) for classification of object status in linear multivariate calibration (MVC) and other data analytical contexts. SIC is a method that directly constructs an interval estimator for the predicted response. SIC is based on the single assumption that all errors involved in MVC are limited. We present the theory of the SIC method and explain its realization by linear programming techniques. The primary SIC consequence is a radically new object classification that can be interpreted using a two-dimensional object status plot (OSP), 'SIC residual vs SIC leverage'. These two new measures of prediction quality are introduced in the traditional chemometric MVC context. Simple straight demarcations divide the OSP into areas which quantitatively discriminate all objects involved in modeling and prediction into four different types: boundary samples, which are the significant objects (for generating the entire data structure) within the training subset; insiders, which are samples that comply with the model; outsiders, which are samples that have large prediction errors; and finally outliers, which are those samples that cannot be predicted at all with respect to a given model. We also present detailed comparisons of the new SIC approach with traditional chemometric methods applied for MVC, classification and outlier detection. These comparisons employ four real-world data sets, selected for their particular complexities, which serve as showcases of SIC application on intricate training and test set data structures.

Chemometrics: achievements and prospects

“…99 The uncertainty in the quantitative analysis is described by either a value, i.e., standard deviation, 100 or an interval, i.e., a confidence 101 or prediction interval. 56 In qualitative analysis, verification of statistical hypotheses is used 102 in which the uncertainty is characterised by the probability of making a wrong decision. 103 The methods for estimation of the uncertainty during modelling of multivariate 104 and multiway 105 data arouse considerable interest in chemometricians.…”

“…The data significance can be estimated by classical regression methods 112 or by non-statistical procedures. 56 When the constructed model is used to determine desired parameters, similar problems arise. The method may prove inapplicable to some samples (an outlier in the prediction 113 ) or give inaccurate results.…”

“…Razumov 54,55 and his colleagues from the Institute of Chemical Physics of the RAS (Chernogolovka) employ multivariate methods of data analysis to solve problems of chemical kinetics. In recent years, new research groups that develop and utilise chemometric methods appeared in Russia, namely, the groups of Rodionova, 56 Pomerantsev, 57 Bogomolov 58,59 (in Moscow); Kucheryavski, 60 Zhilin 61 (in Barnaul); Romanenko 62 (in Tomsk) and Shabanova and Vasil'ev 63 (in Irkutsk).…”

“…In this case, the prediction gives an interval; therefore, this method was called`simple interval calculation'. 36,56 Several works in analytical chemistry have been performed using this method. 166…”

“…188 The purpose of this and other transformations, for example, the Box ± Cox transformation, 33 is model linearisation. Simple operations with the data such as taking the logarithm 56 or extraction of the root 36 can often markedly improve the model. Almost in all cases, the initial data contain errors both random and systematic.…”