Ranks underlie outcome of combining classifiers: Quantitative roles for and

Abstract

Combining classifier systems potentially improves predictive accuracy, but outcomes have proven impossible to predict. Classification most commonly improves when the classifiers are ''sufficiently good'' (generalized as ''ACCURACY'') and ''sufficiently different'' (generalized as ''DIVERSITY''), but the individual and joint quantitative influence of these factors on the final outcome remains unknown. We resolve these issues. Beginning with simulated data, we develop the DIRAC framework (DIVERSITY of Ranks and ACCURACY), which accurately predicts outcome of both score-based fusions originating from exponentially modified Gaussian distributions and rank-based fusions, which are inherently distribution independent. DIRAC was validated using biological dual-energy X-ray absorption and magnetic resonance imaging data. The DIRAC framework is domain independent and has expected utility in far-ranging areas such as clinical biomarker development/personalized medicine, clinical trial enrollment, insurance pricing, portfolio management, and sensor optimization.THE BIGGER PICTURE It can be advantageous to combine multiple predictive models for power or robustness, but it is recognized that realizing these potential gains cannot be guaranteed, especially when the input models cannot be appropriately weighted a priori or the resulting fusion models cannot be cross-validated. This, and a series of mathematically related problems in different guises, fundamentally limits the ability to optimally use all available models to improve classification across essentially all domains in which more than one potentially useful model exists. We show that any fusion's outcome is fully predictable/explicable given characterization of the models to be fused. The ''mechanism'' described acts at the level of ranks, not scores, which extends our findings to all distributions and, functionally, to any domain of interest. We are elucidating the underlying math, following the framework's implications for data science, and using this approach on real-world problems.Concept: Basic principles of a new data science output observed and reported

Improving SDG Classification Precision Using Combinatorial Fusion

“…The results of the current paper are in line with those findings. Namely, the pattern matches the results from three other recent publications [1,15,31] where model fusion and cognitive diversity were used to perform combinatorial fusion. The combined models perform consistently as well as, and in many cases outperform, the best of the two individual models.…”

“…Combinatorial fusion algorithm (CFA) provides methods and algorithms for combining multiple scoring systems using the rank-score characteristic (RSC) function and cognitive diversity (CD) [3,4]. It has been used widely in protein structure prediction [5], ChIP-seq peak detection [6], virtual screening and drug discovery [7,8], target tracking [9], stress detection [10,11], portfolio management [12], visual cognition [13], wireless network handoff detection [14], combining classifiers with diversity and accuracy [15], and text categorization [16], to name just a few (see [17][18][19] and the references within).…”

Predicting drug toxicity at the intersection of informatics and biology: DTox builds a foundation

“…Specifically, a particular interest raised by this study is the potential use of system-level information fusion for drug discovery. It is recognized that diversity between mathematical models can be more important than accuracy for successful fusions, 8 and the importance of diversity in fusion has been specifically demonstrated for in silico approaches based on intact molecules 9 and binding motifs. 10 DTox is conceptually orthogonal to existing in silico modeling approaches; the likely synergy enables improved modeling without requiring additional primary data.…”

The DIRAC framework: Geometric structure underlies roles of and in combining classifiers

“…Our initial paper introducing the DIRAC (diversity of ranks and accuracy) framework contained a review of some current work in this area. 19 …”

“…Worse, the no-free-lunch theorems 20 , 21 , 22 predict that the outcome of system fusion is provably impossible to predict in the absence of context. 19 …”