Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

Liu, Shiwei; Chen, Tianlong; Atashgahi, Zahra; Chen, Xiaohan; Sokar, Ghada; Mocanu, Elena; Pechenizkiy, Mykola; Wang, Zhangyang; Mocanu, Decebal Constantin

doi:10.48550/arxiv.2106.14568

Cited by 5 publications

(8 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Fig. 5, Our method achieves over 1% higher performance in its optimal case (q = 0.5) compared to the best-performing case of MIMO (q = 0.0), specifically when using WRN28- We also measured changes in diversity among individual predictions when varying q, since this diversity in ensemble learning is one of the key factors associated with performance [31], [8], [10], [32].…”

Section: B: Experimental Resultsmentioning

confidence: 99%

“…BatchEnsemble can achieve performance similar to the traditional deep ensemble method with only a small number of additional parameters from the parameterized vectors. Pruning-based approaches decreasing the number of floating-point operations (FLOPs) relative to the traditional deep ensemble method have been also developed [7], [8].…”

Section: B Implicit Deep Ensemblesmentioning

confidence: 99%

“…Recent research has focused on how to efficiently boost performance with mild overhead. One such example is implicit deep ensembles [6], [7], [8], [9], [10], which share the parameters of a single neural network among several ensemble members. Here, a small number of additional parameters for each member is introduced to make those ensemble members non-identical.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient Ensemble via Rotation-Based Self- Supervised Learning Technique and Multi-Input Multi-Output Network

Park

2024

IEEE Access

View full text Add to dashboard Cite

Multi-input multi-output structures have been developed to boost performance by learning multiple ensemble members at a small additional cost to a single network. There were several attempts to further develop multi-input multi-output structures; however, integrating the benefits of self-supervised learning into a multi-input multi-output structure has not yet been studied. In this work, we develop a multiinput multi-output structure designed to jointly learn original and self-supervised tasks, thereby leveraging the benefits of self-supervised learning. Specifically, in terms of multiple inputs, we improve the mixing strategy and minibatch structure for rotation-based self-supervised learning technique, and in terms of multiple outputs, we extend the label space of multiple classifiers to predict both the original class and true rotation degree. We observe that our method with wider networks on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets shows better performance compared to previous works, even with nearly half the number of parameters, e.g., using only about 45.8% of the number of parameters compared to the bestperforming multi-input multi-output method, MixMo, in the Tiny ImageNet dataset, while still achieving a 2.01% improvement. INDEX TERMS convolution neural networks (CNNs), deep ensemble, multi-input multi-output networkThis article has been accepted for publication in IEEE Access.

show abstract

Section: B: Experimental Resultsmentioning

confidence: 99%

Section: B Implicit Deep Ensemblesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Ensemble via Rotation-Based Self- Supervised Learning Technique and Multi-Input Multi-Output Network

Park

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Finally, we would like to note that there is increasing research efforts to develop UQ methods for ML/AI models, which would play important roles in enabling Bayesian UQ paradigm for uncertainty-aware quantication of reproducibility. While detailed presentation of such methods would be outside the scope of this article, we refer interested readers to relevant papers on the uncertainty quantication of ML/AI models, [64][65][66][72][73][74] as well as the references therein.…”

Section: Potential Applications Of Uncertaintyaware Reproducibility M...mentioning

confidence: 99%

A rigorous uncertainty-aware quantification framework is essential for reproducible and replicable machine learning workflows

Pouchard,

Reyes,

Alexander

et al. 2023

Digital Discovery

View full text Add to dashboard Cite

show abstract

“…Although IMP methods [84,2,18,5,46] [24,82,51]. Based on IMP technique, researchers found the existence of LTH in various applications including visual recognition tasks [24], natural language processing [4,7,53], reinforcement learning [69,81], generative model [31], low-cost neural network ensembling [46], and improving robustness [8]. Although LTH has been actively explored in ANN domain, LTH for SNNs is rarely studied.…”

Section: Lottery Ticket Hypothesismentioning

confidence: 99%

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

Kim¹,

Li²,

Park³

et al. 2022

Preprint

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have recently emerged as a new generation of low-power deep neural networks, which is suitable to be implemented on low-power mobile/edge devices. As such devices have limited memory storage, neural pruning on SNNs has been widely explored in recent years. Most existing SNN pruning works focus on shallow SNNs (2∼6 layers), however, deeper SNNs (≥16 layers) are proposed by state-of-the-art SNN works, which is difficult to be compatible with the current SNN pruning work. To scale up a pruning technique towards deep SNNs, we investigate Lottery Ticket Hypothesis (LTH) which states that dense networks contain smaller subnetworks (i.e., winning tickets) that achieve comparable performance to the dense networks. Our studies on LTH reveal that the winning tickets consistently exist in deep SNNs across various datasets and architectures, providing up to 97% sparsity without huge performance degradation. However, the iterative searching process of LTH brings a huge training computational cost when combined with the multiple timesteps of SNNs. To alleviate such heavy searching cost, we propose Early-Time (ET) ticket where we find the important weight connectivity from a smaller number of timesteps. The proposed ET ticket can be seamlessly combined with a common pruning techniques for finding winning tickets, such as Iterative Magnitude Pruning (IMP) and Early-Bird (EB) tickets. Our experiment results show that the proposed ET ticket reduces search time by up to 38% compared to IMP or EB methods. Code is available at Github.

show abstract

Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

Cited by 5 publications

References 27 publications

Efficient Ensemble via Rotation-Based Self- Supervised Learning Technique and Multi-Input Multi-Output Network

Efficient Ensemble via Rotation-Based Self- Supervised Learning Technique and Multi-Input Multi-Output Network

A rigorous uncertainty-aware quantification framework is essential for reproducible and replicable machine learning workflows

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

Contact Info

Product

Resources

About