The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes

Abstract

Understanding the ``fit'' of models meant to predict binary outcomes has been a long-standing problem. We propose a novel metric---the InterModel Vigorish (IMV)---for quantifying the value of change in predictive accuracy between two systems in the case of a binary outcome. The metric is based on an analogy to well-characterized physical systems with tractable probabilities. We first translate a baseline prediction of some binary outcomes into a statement about a canonical system---weighted coins---by equating the entropy of the two systems. We then use the weighted coin for a baseline prediction to establish a fair bet. For a second predictive system that we want to gauge relative to the baseline, we use the notion of expected winnings from a single-blind bet wherein the second weighted coin has replaced the first (the opposing player being blinded to this replacement). The resulting quantity has a scale that is both generally applicable and not dependent on the magnitude of the baseline prediction; moreover, it is always a statement about the change in fit relative to some baseline (which can simply be the prevalence) whereas other metrics (e.g., AUC) are stand-alone measures that need to be further manipulated to yield indices related to differences in fit across models. We illustrate the properties of this metric in simulations, and the value of it in empirical applications related to health, political affiliation, and item responses. We also reconsider results from the recent Fragile Families Challenge using the IMV metric.

Reweighting the UK Biobank to reflect its underlying sampling population substantially reduces pervasive selection bias due to volunteering

“…This is consistent with the notion that individuals of a certain type are more likely to get sampled in the UKB. This pattern of smaller standard deviations is also in line with our simulations as can be visually seen in Our LASSO probit model adequately discriminates between UKB and UK Census observations, with an area under the curve (AUC) of 0.767 [16] (IM V = 0.006 [17]) in the independent holdout sample. For comparison, our AUC is similar to the AUC achieved when predicting mortality in the Health and Retirement Study to correct for mortality selection bias [18].…”

“…Our LASSO probit model adequately discriminates between UKB and UK Census observations, with an area under the curve (AUC) of 0.767 [16] ( IMV = 0.006 [17]) in the independent holdout sample. For comparison, our AUC is similar to the AUC achieved when predicting mortality in the Health and Retirement Study to correct for mortality selection bias [18].…”

Linguistic, cultural, and narrative capital: computational and human readings of transfer admissions essays

“…To account for the data imbalances, we use Welch’s t test and two different but complementary methods within the supervised learning framework. The first is the InterModel Vigorish (IMV), a new technique to analyze the outcomes of predictive models (specifically logistic regression) that works by comparing the predictive accuracy of a given model with another model that only predicts the mode outcome [ 50 ]. The results from this method could be compared to the common vigorish (percentage of winnings deducted from players) of 1% in blackjack which, on average, becomes 0.0099 cents per dollar for the casino.…”

Linguistic, Cultural, and Narrative Capital: Computational and Human Readings of Transfer Admissions Essays

“…For the 2015-2016, the IMV between a baseline prevalence model and one that uses essays was approximately 0.03. This is comparable to the IMV for predicting death within the next two years for those approximately 90 years old given additional information about their age, sex, and education level [44]. In the models for the 2016-2017 essays and entire corpus, the IMV drops to approximately one third of the IMV score for the 2015-2016 essays.…”

“…To account for the data imbalances, we use Welch's t-test and two different but complementary methods within the supervised learning framework. The first is the InterModel Vigorish (IMV), a new technique to analyze the outcomes of predictive models (specifically logistic regression) that works by comparing the predictive accuracy of a given model with another model that only predicts the mode outcome [44]. The results from this method could be compared to the common vigorish (percentage of winnings deducted from players) of 1% in blackjack which, on average, becomes 0.0099 cents per dollar for the casino.…”