Μnas

Liberis, Edgar; Dudziak, Łukasz; Lane, Nicholas D.

doi:10.1145/3437984.3458836

Cited by 53 publications

(10 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Rejection: only feasible solutions are retained during the optimization process, and infeasible solutions are automatically discarded (Dong et al, 2018 ; Hsu et al, 2018 ; Liberis et al, 2021 ).…”

Section: Main Conceptsmentioning

confidence: 99%

“…Within this survey, we illustrate various solutions from the literature aimed at adapting NAS systems for TinyML. Specifically, we examine several NAS frameworks (Dong et al, 2018 ; Zhou et al, 2018 ; Jin et al, 2019 ; Tan et al, 2019 ; Fraccaroli et al, 2021 , 2022 ; Liberis et al, 2021 ) and explore potential methodologies for incorporating physical constraints into synthesized networks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Integration between constrained optimization and deep networks: a survey

Bizzarri,

Fraccaroli,

Lamma

et al. 2024

Front. Artif. Intell.

View full text Add to dashboard Cite

Integration between constrained optimization and deep networks has garnered significant interest from both research and industrial laboratories. Optimization techniques can be employed to optimize the choice of network structure based not only on loss and accuracy but also on physical constraints. Additionally, constraints can be imposed during training to enhance the performance of networks in specific contexts. This study surveys the literature on the integration of constrained optimization with deep networks. Specifically, we examine the integration of hyper-parameter tuning with physical constraints, such as the number of FLOPS (FLoating point Operations Per Second), a measure of computational capacity, latency, and other factors. This study also considers the use of context-specific knowledge constraints to improve network performance. We discuss the integration of constraints in neural architecture search (NAS), considering the problem as both a multi-objective optimization (MOO) challenge and through the imposition of penalties in the loss function. Furthermore, we explore various approaches that integrate logic with deep neural networks (DNNs). In particular, we examine logic-neural integration through constrained optimization applied during the training of NNs and the use of semantic loss, which employs the probabilistic output of the networks to enforce constraints on the output.

show abstract

Section: Main Conceptsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Integration between constrained optimization and deep networks: a survey

Bizzarri,

Fraccaroli,

Lamma

et al. 2024

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…When assembling a large number of networks, it is important to induce different networks to create synergy without conflict. Simply ensemble them all, as in (10), can often cause conflict.…”

Section: B Proposed Ensemble Distillationmentioning

confidence: 99%

“…The ensemble distillation using only the final teachers trained independently without curriculum learning, as shown in (7), showed an accuracy of 76.1% at -12.5 dB. The stage ensemble, which uses not only the final teachers trained independently but also the intermediate result teachers for each stage, as shown in (10), showed an accuracy of 76.9% at -12.5 dB. Finally, the proposed weighted-stage ensemble, which provides weights to the intermediate result teachers for each stage according to the SNR of the training sample, as shown in (11), showed an accuracy of 78.6% at -12.5 dB.…”

Section: Figure 5 Comparison Of the Accuracy Of The Large Network Bet...mentioning

confidence: 99%

“…One of effective techniques is knowledge distillation, which transfers the superior performance of a large teacher network to a small student network [8]. In the previous studies, keyword spotting models based on tiny convolutional networks was implemented on resource-constrained but energy-efficient microcontrollers in real time [9], [10].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Joint Framework of Curriculum Learning and Knowledge Distillation for Noise-Robust and Small-Footprint Keyword Spotting

Lim,

Baek

2023

IEEE Access

View full text Add to dashboard Cite

Spoken keyword spotting has been widely used to facilitate an always-on voice interface in consumer electronics, owing to its simplicity and low latency. Small-footprint keyword spotting based on tiny convolutional neural networks can be implemented on resource-constrained but energy-efficient microcontrollers in real time. However, it is difficult for tiny neural networks to learn the noise-robustness properties essential for successful voice interfaces. To overcome this problem, this study proposes a joint framework of curriculum learning and knowledge distillation for noise-robust small-footprint keyword spotting. The proposed joint framework applies noise mixture curriculum learning to a network that is sufficiently large for learning various noise situations. Subsequently, knowledge distillation is applied to compress the large network into a network small enough to be onboard microcontrollers. To enhance the effectiveness of the joint framework, we propose curriculum learning using a new noise mixture strategy and knowledge distillation with an effective ensemble of neural network snapshots for each curriculum stage. These proposed methods enable the effective learning of noisy situations by large networks and the transfer of noise robustness to small networks. The effectiveness of our joint framework is illustrated using the Google Speech Commands dataset with noise mixtures from various public noise datasets. Our joint framework results in superior performance in noisy situations compared with state-of-the-art noise-robust keyword spotting methods. Therefore, the proposed framework significantly improves the usability of voice interfaces in consumer electronics.

show abstract

Multi-objective Framework for Training and Hardware Co-optimization in FPGAs

Mansoori

Casu

2023

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

Μnas

Cited by 53 publications

References 9 publications

Integration between constrained optimization and deep networks: a survey

Integration between constrained optimization and deep networks: a survey

Joint Framework of Curriculum Learning and Knowledge Distillation for Noise-Robust and Small-Footprint Keyword Spotting

Multi-objective Framework for Training and Hardware Co-optimization in FPGAs

Contact Info

Product

Resources

About