Speeding-up one-versus-all training for extreme classification via mean-separating initialization

Schultheis, Erik; Babbar, Rohit

doi:10.1007/s10994-022-06228-2

Cited by 8 publications

(2 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This way, assuming logits are close to zero when we begin training, the model will assign probability ≈ k n to each label instead of ≈ 1 2 . A similar bias initialisation idea for MLC was discussed in (Schultheis and Babbar 2022), but it was not used in a neural network.…”

Section: Model Setupmentioning

confidence: 99%

Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

Grivas,

Vergari,

Lopez

2024

AAAI

View full text Add to dashboard Cite

Sigmoid output layers are widely used in multi-label classification (MLC) tasks, in which multiple labels can be assigned to any input. In many practical MLC tasks, the number of possible labels is in the thousands, often exceeding the number of input features and resulting in a low-rank output layer. In multi-class classification, it is known that such a low-rank output layer is a bottleneck that can result in unargmaxable classes: classes which cannot be predicted for any input. In this paper, we show that for MLC tasks, the analogous sigmoid bottleneck results in exponentially many unargmaxable label combinations. We explain how to detect these unargmaxable outputs and demonstrate their presence in three widely used MLC datasets. We then show that they can be prevented in practice by introducing a Discrete Fourier Transform (DFT) output layer, which guarantees that all sparse label combinations with up to k active labels are argmaxable. Our DFT layer trains faster and is more parameter efficient, matching the F1@k score of a sigmoid layer while using up to 50% fewer trainable parameters. Our code is publicly available at https://github.com/andreasgrv/sigmoid-bottleneck.

show abstract

Section: Model Setupmentioning

confidence: 99%

Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

Grivas,

Vergari,

Lopez

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…However, further study is required to analyze its theoretical features, optimize hyperparameters, and test its scalability to more expansive networks. [19] presents an approach to accelerate deep learning training using adaptive conjugate gradient (ACG) optimization. The paper introduces an ACG optimization algorithm for deep learning training, incorporating adaptive learning rates and conjugate gradient directions to update network weights.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Stochastic Conjugate Gradient Optimization for Backpropagation Neural Networks

Hashem,

Alaba,

Jumare

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Backpropagation neural networks are commonly utilized to solve complicated issues in various disciplines. However, optimizing their settings remains a significant task. Traditional gradient-based optimization methods, such as stochastic gradient descent (SGD), often exhibit slow convergence and hyperparameter sensitivity. An adaptive stochastic conjugate gradient (ASCG) optimization strategy for backpropagation neural networks is proposed in this research. ASCG combines the advantages of stochastic optimization and conjugate gradient techniques to increase training efficiency and convergence speed. Based on the observed gradients, the algorithm adaptively calculates the learning rate and search direction at each iteration, allowing for quicker convergence and greater generalization. Experimental findings on benchmark datasets show that ASCG optimization outperforms standard optimization techniques regarding convergence time and model performance. The proposed ASCG algorithm provides a viable method for improving the training process of backpropagation neural networks, making them more successful in tackling complicated problems across several domains. As a result, the information for initial seeds formed while the model is being trained grows. The coordinated efforts of ASCG's Conjugate Gradient and ASCG components improve learning and achieve global minima. Our results indicate that our ASCG algorithm achieves 21 percent higher accuracy on the HMT dataset and performs better than existing methods on other datasets(DIR-Lab dataset). The experimentation revealed that the conjugate gradient has an efficiency of 95 percent when utilizing the principal component analysis features, compared to 94 percent when using the correlation heatmap features selection approach with MSE of 0.0678.

show abstract

Towards Memory-Efficient Training for Extremely Large Output Spaces – Learning with 670k Labels on a Single Commodity GPU

Schultheis,

Babbar

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory. Using sparse connectivity would drastically reduce the memory requirements, but as we show below, applied naïvely it can result in much diminished predictive performance. Fortunately, we found that this can be mitigated by introducing an intermediate layer of intermediate size. We further demonstrate that one can constrain the connectivity of the sparse layer to be of constant fan-in, in the sense that each output neuron will have the exact same number of incoming connections, which allows for more efficient implementations, especially on GPU hardware. The CUDA implementation of our approach is provided at https://github.com/xmc-aalto/ecml23-sparse.

show abstract

Speeding-up one-versus-all training for extreme classification via mean-separating initialization

Cited by 8 publications

References 26 publications

Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

Adaptive Stochastic Conjugate Gradient Optimization for Backpropagation Neural Networks

Towards Memory-Efficient Training for Extremely Large Output Spaces – Learning with 670k Labels on a Single Commodity GPU

Contact Info

Product

Resources

About