Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities

Abstract

In recent times, deep neural networks achieved outstanding predictive performance on various classification and pattern recognition tasks. However, many real-world prediction problems have ordinal response variables, and this ordering information is ignored by conventional classification losses such as the multi-category cross-entropy. Ordinal regression methods for deep neural networks address this. One such method is the CORAL method, which is based on an earlier binary label extension framework and achieves rank consistency among its output layer tasks by imposing a weight-sharing constraint. However, while earlier experiments showed that CORAL's rank consistency is beneficial for performance, the weight-sharing constraint could severely restrict the expressiveness of a deep neural network. In this paper, we propose an alternative method for rank-consistent ordinal regression that does not require a weight-sharing constraint in a neural network's fully connected output layer. We achieve this rank consistency by a novel training scheme using conditional training sets to obtain the unconditional rank probabilities through applying the chain rule for conditional probability distributions. Experiments on various datasets demonstrate the efficacy of the proposed method to utilize the ordinal target information, and the absence of the weight-sharing restriction improves the performance substantially compared to the CORAL reference approach.

Prediction of Small-Molecule Developability Using Large-Scale In Silico ADMET Models

“…Hyperparameters were optimized similarly to the neural network and yielded a tree depth of 1 (deeper trees generalized worse, likely due to overfitting). One reason for the better performance of the neural network might be the various techniques employed to deal with overfitting such as early stopping, 46 dropout, 47 and rank-consistent ordinal regression 25,26 (for details, see Section 4), which is a major problem in the presented setting due to the small number of positive examples.…”

“…The neural network was implemented in PyTorch 50 using the CORN loss for rank-consistent ordinal regression. 25,26 As an optimizer we chose AdamW 51 and trained the network using early stopping 46 and cyclical learning rates. 52 For the cyclical learning rate scheduling, we chose a base learning rate of 0.0001, a maximum learning rate of 0.01, the "triangular2" cycling mode, and the number of iterations of one cycle was set to four times the batch size.…”

“…Using the above, we trained a deep neural network employing rank-consistent ordinal regression 25,26 to infer the potential of progression of compounds beyond in vivo PK studies (for details, see Section 4.2) based on the predicted ADMET profile of the compounds (Figure 1a). A major problem of training machine learning models on the data set is its high imbalance: compounds that progressed beyond in vivo PK are very rare (Figure S1, left).…”

Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation

“…They reported that the CORAL framework provided both rank consistency and superior results compared to the previous approaches. Shi et al [24] proposed Conditional Ordinal regression for Neural Network (CORN) framework to relax the constraint on the penultimate layer of the CORAL framework to increase neural network's capacity by introducing conditional probabilities. The authors reported that the CORN approach performs better than previous methods.…”

“…In addition, we have experimented with CORN framework, which is designed for the ordinal regression tasks. CORN framework has recently been introduced by Shi et al [24] and it is an improvement on previous ordinal regression approaches that transform the task into a series of binary classification sub-problems [14,4]. Experimental results show that the CORN approach and its derivatives (OR-NN [14] and CORAL [4]), have superior results compared to other ordinal regression approaches.…”