Sum of ranking differences for method discrimination and its validation: comparison of ranks with random numbers y Ká roly Hé berger a * and Klá ra Kollá r-Hunek b This paper describes the theoretical background, algorithm and validation of a recently developed novel method of ranking based on the sum of ranking differences [TrAC Trends Anal. Chem. 2010; 29: 101-109]. The ranking is intended to compare models, methods, analytical techniques, panel members, etc. and it is entirely general. First, the objects to be ranked are arranged in the rows and the variables (for example model results) in the columns of an input matrix. Then, the results of each model for each object are ranked in the order of increasing magnitude. The difference between the rank of the model results and the rank of the known, reference or standard results is then computed. (If the golden standard ranking is known the rank differences can be completed easily.) In the end, the absolute values of the differences are summed together for all models to be compared. The sum of ranking differences (SRD) arranges the models in a unique and unambiguous way. The closer the SRD value to zero (i.e. the closer the ranking to the golden standard), the better is the model. The proximity of SRD values shows similarity of the models, whereas large variation will imply dissimilarity. Generally, the average can be accepted as the golden standard in the absence of known or reference results, even if bias is also present in the model results in addition to random error. Validation of the SRD method can be carried out by using simulated random numbers for comparison (permutation test). A recursive algorithm calculates the discrete distribution for a small number of objects (n < 14), whereas the normal distribution is used as a reasonable approximation if the number of objects is large. The theoretical distribution is visualized for random numbers and can be used to identify SRD values for models that are far from being random. The ranking and validation procedures are called Sum of Ranking differences (SRD) and Comparison of Ranks by Random Numbers (CRNN), respectively.

No abstract

The old debate is revived: Definite differences can be observed in suggestions of estimation for prediction performances of models and for validation variants according to the various scientific disciplines. However, the best and/or recommended practice for the same data set cannot be dependent on the field of usage. Fortunately, there is a method comparison algorithm, which can rank and group the validation variants; its combination with variance analysis will reveal whether the differences are significant or merely the play of random errors. Therefore, three case studies have been selected carefully to reveal similarities and differences in validation variants. The case studies illustrate the different significance of these variants well. In special circumstances, any of the influential factors for validation variants can exert significant influence on evaluation by sums of (absolute) ranking differences (SRDs): stratified (contiguous block) or repeated Monte Carlo resampling and how many times the data set is split (5‐7‐10). The optimal validation variant should be determined individually again and again. A random resampling with sevenfold cross‐validations seems to be a good compromise to diminish the bias and variance alike. If the data structure is unknown, a randomization of rows is suggested before SRD analysis. On the other hand, the differences in classifiers, validation schemes, and models proved to be always significant, and even subtle differences can be detected reliably using SRD and analysis of variance (ANOVA).

No abstract

aThe analysis of descriptive sensory data is a very complex task in food product development and in food quality management. One of the key issues is the reliability of the panel members in making decisions. The authors created a software -the ProfiSens, based on Visual Basic for Excel (VBE) -for the IT support of food profile analysis. Using this software we have developed a method for testing panel consistency. This method is based on the geometrical properties of the profile polygon. It offers a fast and expressive way for the qualification of the panel, and the authors created its VBE implementation, too, that can be easily linked to the basic ProfiSens software. In this paper we discuss the mathematical background and the algorithm of the method in every detail, and we show several applications in the fields of research, education and industry.