Tracking the risk of a deployed model and detecting harmful distribution shifts

Abstract

When deployed in the real world, machine learning models inevitably encounter changes in the data distribution, and certain-but not all-distribution shifts could result in significant performance degradation. In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially, making interventions by a human expert (or model retraining) unnecessary. While several works have developed tests for distribution shifts, these typically either use non-sequential methods, or detect arbitrary shifts (benign or harmful), or both. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate. In this work, we design simple sequential tools for testing if the difference between source (training) and target (test) distributions leads to a significant drop in a risk function of interest, like accuracy or calibration. Recent advances in constructing time-uniform confidence sequences allow efficient aggregation of statistical evidence accumulated during the tracking process. The designed framework is applicable in settings where (some) true labels are revealed after the prediction is performed, or when batches of labels become available in a delayed fashion. We demonstrate the efficacy of the proposed framework through an extensive empirical study on a collection of simulated and real datasets.

Constructing Certainty in Machine Learning: On the performativity of testing and its hold on the future

“…Considering the problems with the accuracy metrics discussed in the previous section, it is also encouraging that there are various measures taken within the ML community to address issues. For example, continuous performance monitoring seeks to identify and fix deteriorating performance over time (Podkopaev and Ramdas 2022), uncertainty quantification provides insights into unknowns in testing (Siddique et al 2022), and methods for interpretability can aid in identifying spurious and problematic correlations that inflate accuracy numbers (Poechhacker and Kacianka 2021). In particular, external auditing is a promising quantitative approach for constructing different, partial accounts of opaque algorithms that often center perspectives and experiences of marginalized people to make possible harms visible (Sandvig et al 2014;Metaxa et al 2021).…”