Circular classifications are classification scales with categories that exhibit a certain periodicity. Since linear scales have endpoints, the standard weighted kappas used for linear scales are not appropriate for analyzing agreement between two circular classifications. A family of kappa coefficients for circular classifications is defined. The kappas differ only in one parameter. It is studied how the circular kappas are related and if the values of the circular kappas depend on the number of categories. It turns out that the values of the circular kappas can be strictly ordered in precisely two ways. The orderings suggest that the circular kappas are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for circular kappas that tend to produce higher values.

No abstract

No abstract

When validating a new test, its incremental predictive validity (IV) is often evaluatedfrom a classical approach using in-sample estimates. In this study, to evaluate IV, weadopt a statistical learning (SL) approach that uses out-of-sample estimates. Followingthis approach, we identify two important aspects to consider: 1) evaluate the predictionrules based on the out-of-sample prediction error, and 2) obtain the prediction rulesusing different estimation methods (e.g., OLS, SIMEX, and ridge). We performed asimulation study to investigate the effect of using in- and out-of-sample measures on IV,and to assess the influence of the reliabilities of tests on these two estimates of IV.Results showed that the two approaches differed, such that in-sample IV is on averagelarger than out-of-sample IV, but that the difference decreases as the size of thecalibration sample increases. In- and out-of-sample IV were both affected by thereliability of the added test, that is, as reliability increases IV also increases. Inaddition, if collinearity exists between two tests, the reliability of the first test has anegative relationship with the IV of the second test. Worthy of note is that thedirection of these effects did not differ between the two approaches (in/out-of-sample).It was further shown that using ridge regression enhances IV defined as out-of-samplewhen the size of the calibration sample is small. We also showed how to apply the newprinciples in practice using empirical data from a student selection procedure. Weconcluded that in addition to establishing strong reliability for the new test, validationshould be based on out-of-sample prediction accuracy and that users should consider ashrinkage method to estimate optimal rules for prediction for new persons.