Discriminative Metric Learning with Deep Forest

Abstract

A Discriminative Deep Forest (DisDF) as a metric learning algorithm is proposed in the paper. It is based on the Deep Forest or gcForest proposed by Zhou and Feng and can be viewed as a gcForest modification. The case of the fully supervised learning is studied when the class labels of individual training examples are known. The main idea underlying the algorithm is to assign weights to decision trees in random forest in order to reduce distances between objects from the same class and to increase them between objects from different classes. The weights are training parameters. A specific objective function which combines Euclidean and Manhattan distances and simplifies the optimization problem for training the DisDF is proposed. The numerical experiments illustrate the proposed distance metric algorithm. of supervised distance metric learning is cast into pairwise constraints: the equivalence constraints where pairs of data points that belong to the same classes, and inequivalence constraints where pairs of data points belong to different classes.Metric learning approaches were reviewed in [1,5,14,29]. The basic idea underlying the metric learning solution is that the distance between similar objects should be smaller than the distance between different objects. If we have two observation vectors x i ∈ R m and x j ∈ R m from a training set, and the similarity of objects is defined by their belonging to the same class, then the distance d(x i , x j ) between the vectors should be minimized if x i and x j belong to the same class, and it should be maximized if x i and x j are from different classes. Several review papers analyze various methods and algorithms of metric learning [12,19,27]. A powerful implementation of the metric learning dealing with non-linear data structures is the so-called Siamese neural network introduced by Bromley et al. [4] in order to solve signature verification as a problem of image matching. This network consists of two identical sub-networks joined at their outputs. The two sub-networks extract features from two input examples during training, while the joining neuron measures the distance between the two feature vectors. The Siamese architecture has been exploited in many applications, for example, in face verification [7], in the one-shot learning in which predictions are made given only a single example of each new class [13], in constructing an inertial gesture classification [2], in deep learning [24], in extracting speaker-specific information [6], for face verification in the wild [11]. This is only a part of successful applications of Siamese neural networks. Many modifications of Siamese networks have been developed, including fully-convolutional Siamese networks [3], Siamese networks combined with a gradient boosting classifier [15], Siamese networks with the triangular similarity metric [29].A new powerful method, which can be viewed as an alternative to deep neural networks, is the deep forest proposed by Zhou and Feng [30] and called the gcForest. It can be compared with ...

Improvement of the Deep Forest Classifier by a Set of Neural Networks

“…It has been mentioned that the use of the weighted averaging significantly improves the DF and allows us to solve various machine learning problems by controlling the objective function for computing optimal weights [37,38]. However, we need a more complex function of the class probability distributions sometimes in order to get superior results.…”

“…Some improvements have been proposed by Utkin and Ryabinin [37,38,39]. In particular, modifications of the DF for solving the weakly supervised and fully supervised metric learning problems were proposed in [39] and [37], respectively. A transfer learning model using the DF was presented in [38].…”

Developing Sustainable Classification of Diseases via Deep Learning and Semi-Supervised Learning

“…Such models can achieve excellent performance with the default settings, even if data from distinct domains are considered. Many studies of deep forest methods have been developed [33,34], and these methods have been successfully used in image retrieval [35], and cancer subtype classification [36].…”

Object-Based Multigrained Cascade Forest Method for Wetland Classification Using Sentinel-2 and Radarsat-2 Imagery

“…This method with fewer hyperparameters can automatically determine the model complexity by using input data [62,63]. The gcForest can be able to achieve satisfactory results with small-scale training data in image classification and target identification [64][65][66][67][68]. The potential application ability of gcForest for wetland land cover classification, however, has a shortage of evaluation.…”