Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks

Chin, Ting-Wu; Morcos, Ari S.; Marculescu, Diana

doi:10.1007/978-3-030-86523-8_8

Cited by 12 publications

(7 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another approach for data augmentation is through synthetic data generation using generative models [41]- [43]. In addition to these, some conditional computation approaches strive to improve the model performance with special routing techniques such as using information gain [15] or knowledge distillation from a larger model [44], [45]. Adding multi-path routing approaches to the CIGT technique can be considered in the latter group.…”

Section: Related Work a Conditional Computingmentioning

confidence: 99%

Multi-Path Routing for Conditional Information Gain Trellis Using Cross-Entropy Search and Reinforcement Learning

Bicici,

Akarun

2024

IEEE Access

View full text Add to dashboard Cite

Convolutional neural networks have made significant strides in solving computer-vision tasks at the expense of high computational demands. This complexity hinders efficient processing, particularly on devices with limited computational resources such as edge devices. One way to overcome this limitation is conditional computing, which optimizes inference by selectively utilizing parts of the network depending on the characteristics of the input. A recent conditional execution method is Conditional Information Gain Trellis (CIGT), which routes samples based on an information gain-based router mechanism. The original CIGT model was designed to route a single sample along a single path in a trellis structure. In this study, advanced inference strategies that allow inputs to traverse multiple paths are proposed to improve the performance of the vanilla CIGT model. These strategies aim to find a middle ground between improved model performance and increased computational demands. For this purpose, two techniques were proposed: A Cross-Entropy Search-based threshold optimization algorithm and a Reinforcement Learningbased routing strategy. The first method treats multi-path routing in CIGT as a black-box optimization problem and the second interprets it as a Markov Decision Process with a Q-Learning-based supervised regression algorithm designed as the solution. It has been shown that both of these methods provide significant performance improvements compared to the original CIGT model, with an adjustable increase in the computation. Experiments were conducted on two image datasets, with additional statistical tests and analyses to inspect the behaviors of the proposed algorithms. The novel methods designed in this study for multi-path routing show potential for both the original CIGT model and similar conditional computation approaches that use specific routing mechanisms for selecting network parts based on samples.

show abstract

Section: Related Work a Conditional Computingmentioning

confidence: 99%

Multi-Path Routing for Conditional Information Gain Trellis Using Cross-Entropy Search and Reinforcement Learning

Bicici,

Akarun

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Setting 2: Pruning from scratch. In this setting, the network is trained from scratch [4,6,32,40,58,59]. During each mini-batch iteration, sub-networks in the allowable channel configuration space in Sec.…”

Section: Propertymentioning

confidence: 99%

“…5. After the training, optimized searching method is used to seek the candidate networks [4,6,58]. A recent work also incorporates the searching phase into the training phase by penalizing parameters in the rebuilt network, achieving faster convergence [32].…”

Section: Propertymentioning

confidence: 99%

Revisiting Random Channel Pruning for Neural Network Compression

Adamczewski²,

Wen

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Channel (or 3D filter) pruning serves as an effective way to accelerate the inference of neural networks. There has been a flurry of algorithms that try to solve this practical problem, each being claimed effective in some ways. Yet, a benchmark to compare those algorithms directly is lacking, mainly due to the complexity of the algorithms and some custom settings such as the particular network configuration or training procedure. A fair benchmark is important for the further development of channel pruning.Meanwhile, recent investigations reveal that the channel configurations discovered by pruning algorithms are at least as important as the pre-trained weights. This gives channel pruning a new role, namely searching the optimal channel configuration. In this paper, we try to determine the channel configuration of the pruned models by random search. The proposed approach provides a new way to compare different methods, namely how well they behave compared with random pruning. We show that this simple strategy works quite well compared with other channel pruning methods. We also show that under this setting, there are surprisingly no clear winners among different channel importance evaluation methods, which then may tilt the research efforts into advanced channel configuration searching methods. Code will be released at https: //github.com/ofsoundof/random_channel_ pruning.

show abstract

“…A complexity measure can also be defined in terms of the "model size"; therefore it depends of the target ML algorithm to be optimized (e.g., the number of neurons in one layer (Juang & Hsu, 2014), the number of support vectors in a SVM (Bouraoui et al, 2018), the DNN file size (Shinozaki et al, 2020) or the ensemble size (Garrido & Hernández, 2019) for ensemble algorithms). Alternatively, the number of floating point operations (FLOPS) can be used (Chin et al, 2020;Elsken et al, 2019;Lu et al, 2020;Wang et al, 2019Wang et al, , 2020. This metric is also used to reflect the energy consumption (Han, Pool, Tran, & Dally, 2015), and used along the number of parameters in the network (Smithson et al, 2016).…”

Section: Description Referencesmentioning

confidence: 99%

“…Clearly, the number of costly function evaluations in a typical metamodel-based optimization is much lower than in a metaheuristics-based algorithm, as usually only a single new solution is evaluated in each iteration. Nevertheless, the metamodelbased algorithms by Chin et al (2020) (row 7 in Table 6) require surprisingly many evaluations since the authors dedicate additional evaluations to further improve the solutions found. The metaheuristic-based algorithm by Pathak et al (2020) (row 9 in Table 6) can be particularly expensive, as it performs a chaotic local search to generate N additional solutions for each solution present in the population of a given iteration.…”

Section: Conclusion and Research Opportunitiesmentioning

confidence: 99%

A survey on multi-objective hyperparameter optimization algorithms for Machine Learning

Morales-Hernández¹,

Nieuwenhuyse²,

Rojas³

2021

Preprint

View full text Add to dashboard Cite

Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared which focus on optimizing multiple conflicting objectives simultaneously. This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms, distinguishing between metaheuristicbased algorithms, metamodel-based algorithms and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures, and present future research directions.

show abstract

Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks

Cited by 12 publications

References 27 publications

Multi-Path Routing for Conditional Information Gain Trellis Using Cross-Entropy Search and Reinforcement Learning

Multi-Path Routing for Conditional Information Gain Trellis Using Cross-Entropy Search and Reinforcement Learning

Revisiting Random Channel Pruning for Neural Network Compression

A survey on multi-objective hyperparameter optimization algorithms for Machine Learning

Contact Info

Product

Resources

About