Another Look at Distance-Weighted Discrimination

Abstract

Summary. Distance-weighted discrimination (DWD) is a modern margin-based classifier with an interesting geometric motivation. It was proposed as a competitor to the support vector machine (SVM). Despite many recent references on DWD, DWD is far less popular than the SVM, mainly because of computational and theoretical reasons. We greatly advance the current DWD methodology and its learning theory. We propose a novel thrifty algorithm for solving standard DWD and generalized DWD, and our algorithm can be several hundred times faster than the existing state of the art algorithm based on second-order cone programming. In addition, we exploit the new algorithm to design an efficient scheme to tune generalized DWD. Furthermore, we formulate a natural kernel DWD approach in a reproducing kernel Hilbert space and then establish the Bayes risk consistency of the kernel DWD by using a universal kernel such as the Gaussian kernel. This result solves an open theoretical problem in the DWD literature. A comparison study on 16 benchmark data sets shows that data-driven generalized DWD consistently delivers higher classification accuracy with less computation time than the SVM.

A Corrected Tensor Nuclear Norm Minimization Method for Noisy Low-Rank Tensor Completion

“…In this section, we develop an sGS-ADMM [11,33] to solve the dual of the CTNN model in (3.2). Extensive numerical experiments in many fields demonstrate that the sGS-ADMM is not only convergent but also faster than the possibly nonconvergent directly extended multiblock ADMM and its many other variants, e.g., see [3,11,12,30,33,59] and references therein.…”

Covariance-engaged Classification of Sets via Linear Programming

“…For comparisons, we also supply the estimated β 0 , β and ∇ from the CLIPS procedure to a QDA classifier which is applied to all the observations in a testing set, followed by a majority voting scheme (labeled as QDA-MV). Lastly, we calculate the sample mean and variance of each variable in an observation set to form a new feature vector as done in Miedema et al (2012); then support vector machine (SVM; Cortes and Vapnik, 1995) and distance weighted discrimination (DWD; Marron et al, 2007;Wang and Zou, 2018) are applied to the features to make predictions (labeled as SVM and DWD respectively). We use R library clime to calculate the CLIME estimates, R library e1071 to calculate the SVM classifier, and R library sdwd (Wang and Zou, 2016) to calculate the DWD classifier.…”

Sparse Multicategory Generalized Distance Weighted Discrimination in Ultra-High Dimensions

“…The slack variable η i is introduced to ensure that the corresponding margin d i is non-negative and the constant c > 0 is a tuning parameter to control the overlap between classes. Problem (1) can also be written in a loss-plus-penalty form (e.g., [12]) as…”

“…In binary classification problems, linear SVMs seek a hyperplane maximizing the smallest margin for all data points, while DWD seeks a hyperplane minimizing the sum of inverse margins over all data points. Reference [ 8 ] suggests replacing the inverse margins by the q -th power of the inverse margins in a generalized DWD method; see [ 12 ] for a detailed description. Formally, for a training data set of N observations, where and , binary generalized linear DWD seeks a proper separating hyperplane through the optimization problem where a and are the intercept and slope parameters, respectively.…”