Sum of ranking differences for method discrimination and its validation: comparison of ranks with random numbers

Abstract

Sum of ranking differences for method discrimination and its validation: comparison of ranks with random numbers y Ká roly Hé berger a * and Klá ra Kollá r-Hunek b This paper describes the theoretical background, algorithm and validation of a recently developed novel method of ranking based on the sum of ranking differences [TrAC Trends Anal. Chem. 2010; 29: 101-109]. The ranking is intended to compare models, methods, analytical techniques, panel members, etc. and it is entirely general. First, the objects to be ranked are arranged in the rows and the variables (for example model results) in the columns of an input matrix. Then, the results of each model for each object are ranked in the order of increasing magnitude. The difference between the rank of the model results and the rank of the known, reference or standard results is then computed. (If the golden standard ranking is known the rank differences can be completed easily.) In the end, the absolute values of the differences are summed together for all models to be compared. The sum of ranking differences (SRD) arranges the models in a unique and unambiguous way. The closer the SRD value to zero (i.e. the closer the ranking to the golden standard), the better is the model. The proximity of SRD values shows similarity of the models, whereas large variation will imply dissimilarity. Generally, the average can be accepted as the golden standard in the absence of known or reference results, even if bias is also present in the model results in addition to random error. Validation of the SRD method can be carried out by using simulated random numbers for comparison (permutation test). A recursive algorithm calculates the discrete distribution for a small number of objects (n < 14), whereas the normal distribution is used as a reasonable approximation if the number of objects is large. The theoretical distribution is visualized for random numbers and can be used to identify SRD values for models that are far from being random. The ranking and validation procedures are called Sum of Ranking differences (SRD) and Comparison of Ranks by Random Numbers (CRNN), respectively.

Multivariate assessment of lipophilicity scales—computational and reversed phase thin-layer chromatographic indices

“…different methods for the measurement/calculation of the same property (in this case, lipophilicity measures) [57,58]. It takes a matrix as its input, which contains the samples/molecules in its rows and variables/methods in its columns -thus, a cell in row i and column j contains the property (here, lipophilicity) value calculated/measured for the ith molecule with the jth method.…”

Sum of ranking differences (SRD) to ensemble multivariate calibration model merits for tuning parameter selection and comparing calibration methods

“…The SRD algorithm is a simple, powerful, general process to determine similarities between variables by ranking the variables (columns of the SRD input matrix) across objects (rows of the SRD input matrix) relative to respective object reference (target) ranking values [32][33][34][35][36]. The method is well described in the literature and hence, only briefly outlined here.…”

“…The validation involves determining if the SRD rankings are no different than random rankings [33]. The process is named the comparison of ranks by random numbers (CRRN).…”

“…Random numbers are used for a small number of objects (less than 13, or 9 if ties are present) and the normal distribution is used as the approximate for a large number of objects (13 or greater). The CRRN process is not the validation focus in this paper and the reader is referred to reference [33] for the details of CRRN.…”

“…This paper shows that the sum of ranking differences (SRD) [32][33][34][35][36] is a simple objective process to ensemble multiple model merits for ranking models (tuning parameters) allowing automatic selection of a consensus model or set of models. When CV is used to generate model merits, then SRD allows the models merits computed on each data split to be evaluated, not just the mean values as in the standard CV process to select a tuning parameter.…”