Stability of decision trees and logistic regression

Abstract

Decision trees and logistic regression are one of the most popular and well-known machine learning algorithms, frequently used to solve a variety of real-world problems. Stability of learning algorithms is a powerful tool to analyze their performance and sensitivity and subsequently allow researchers to draw reliable conclusions. The stability of these two algorithms has remained obscure. To that end, in this paper, we derive two stability notions for decision trees and logistic regression: hypothesis and pointwise hypothesis stability. Additionally, we derive these notions for L 2 -regularized logistic regression and confirm existing findings that it is uniformly stable. We show that the stability of decision trees depends on the number of leaves in the tree, i.e., its depth, while for logistic regression, it depends on the smallest eigenvalue of the Hessian matrix of the crossentropy loss. We show that logistic regression is not a stable learning algorithm. We construct the upper bounds on the generalization error of all three algorithms. Moreover, we present a novel stability measuring framework that allows one to measure the aforementioned notions of stability. The measures are equivalent to estimates of expected loss differences at an input example and then leverage bootstrap sampling to yield statistically reliable estimates. Finally, we apply this framework to the three algorithms analyzed in this paper to confirm our theoretical findings and, in addition, we discuss the possibilities of developing new training techniques to optimize the stability of logistic regression, and hence decrease its generalization error.

Multi‐split conformal prediction via Cauchy aggregation

“…All the methods are conducted using the target level α as input. Although CV+ only has a 1 À 2α finite sample guarantee, its empirical coverage is approximately 1 À α under most settings and thus using α rather than α=2 is the suggestion of Barber et al (2021). We perform 100 independent repetitions for all settings and datasets to obtain the results shown in this section.…”

“…Thus, various learning algorithms are known to enjoy mean-square stability. For example, k-nearest neighbor (Devroye & Wagner, 1979a), support vector machines (Bousquet & Elisseeff, 2002) and bagging methods (Arsov et al, 2019) all have mean-square stability with γ n ¼ oð1=nÞ.…”

Preparation of Glycan Arrays Using Glycopeptides Derived From Biomaterials

“…and investigate logistic regression under proportionally high-dimensional settings via the approximate message passing theory, show the biased behaviours of the ordinary maximum likelihood estimation, and propose procedures for debiased estimation and hypothesis testing. Arsov et al (2019) study stability and generalization errors for some models including logistic regression. Salehi et al (2019) investigate the impact of some regularization schemes on logistic regression under proportionally high-dimensional settings via the convex Gaussian min-max theorem.…”