Multi-Task Learning for Multi-Objective Evolutionary Neural Architecture Search

Cai, Ronghong; Luo, Jianping

doi:10.1109/cec45853.2021.9504721

Cited by 21 publications

(58 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Population-based MOO methods [26] mainly include dominance-based [27], [28], decomposition-based [29], [30], indicator-based [31], [32], hybrid-based [33], [34], and modelbased methods [35]- [37]. Due to their easy scalability and gradient-free properties, these population-based methods are widely used in various machine learning problems, such as neuroevolution [38], [39], NAS [1], [40]- [42], feature selection [43], [44], reinforcement learning [45], federated learning [46], [47], MTL [48], and fairness learning [49]. In addition, many surrogate-assisted multi-objective evolutionary algorithms have been proposed to solve those machine learning systems with expensive optimization objectives [50]- [52].…”

Section: Moomentioning

confidence: 99%

Bi-Level Multiobjective Evolutionary Learning: A Case Study on Multitask Graph Neural Topology Search

Wang

Jiao

Zhao

et al. 2024

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

The construction of machine learning models involves many bi-level multi-objective optimization problems (BL-MOPs), where upper level (UL) candidate solutions must be evaluated via training weights of a model in the lower level (LL). Due to the Pareto optimality of sub-problems and the complex dependency across UL solutions and LL weights, an UL solution is feasible if and only if the LL weight is Pareto optimal. It is computationally expensive to determine which LL Pareto weight in the LL Pareto weight set is the most appropriate for each UL solution. This paper proposes a bi-level multi-objective learning framework (BLMOL), coupling the above decisionmaking process with the optimization process of the UL-MOP by introducing LL preference r. Specifically, the UL variable and r are simultaneously searched to minimize multiple UL objectives by evolutionary multi-objective algorithms. The LL weight with respect to r is trained to minimize multiple LL objectives via gradient-based preference multi-objective algorithms. In addition, the preference surrogate model is constructed to replace the expensive evaluation process of the UL-MOP. We consider a novel case study on multi-task graph neural topology search. It aims to find a set of Pareto topologies and their Pareto weights, representing different trade-offs across tasks at UL and LL, respectively. The found graph neural network is employed to solve multiple tasks simultaneously, including graph classification, node classification, and link prediction. Experimental results demonstrate that BLMOL can outperform some state-of-the-art algorithms and generate well-representative UL solutions and LL weights.

show abstract

Section: Moomentioning

confidence: 99%

Bi-Level Multiobjective Evolutionary Learning: A Case Study on Multitask Graph Neural Topology Search

Wang

Jiao

Zhao

et al. 2024

IEEE Trans. Evol. Computat.

View full text Add to dashboard Cite

show abstract

“…2). On the other hand, neural architecture search (NAS) studies up until recently have focused mostly on either image classification problems [1,7,29,33,39,62] or learning tasks in isolation [19,34,54,67]. Few have explored architecture search for joint training of dense prediction tasks.…”

Section: Background and Related Workmentioning

confidence: 99%

“…latency, memory, and energy, alongside prediction accuracy, to guide the search. However, current studies often focus on image classification [1,7,29,33,39,62] or learning tasks in isolation [54,67]. However, performing multiple dense prediction tasks simultaneously can have significant benefits for both inference speed and accuracy since tasks can leverage each other's training signals as inductive biases to improve their own learning and the model's generalization [8].…”

Section: Background and Related Workmentioning

confidence: 99%

“…On the other hand, NAS methods aim to automatically learn an optimal set of neural components and their connections. However, the current 1 short for "Edge-Efficient Dense Predictions via Multi-Task NAS" literature often focuses on either simpler tasks such as classification [7,33,62] or single-task training setup [19,34]. In contrast, we jointly learn MTL-DP and NAS and leverage their strengths to tackle the aforementioned issues simultaneously, resulting in a novel and improved approach to efficient dense predictions for edge.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural Architecture Search

Vu¹,

Zhou²,

Wen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Deep neural networks (DNNs) are becoming ubiquitous across a plethora of intelligent embedded applications such as virtual reality (VR) [1] and object detection/tracking [2], enabling entirely new ondevice experiences [3]. Nonetheless, given that the network design space is tremendously large [4,5], manually designing competitive DNNs requires considerable human efforts to determine the optimal network configuration. To address this, neural architecture search (NAS) [6] has recently flourished, which is dedicated to automating the design of top-performing DNNs.…”

Section: Introductionmentioning

confidence: 99%

You only search once

Luo

Liu

Kong

et al. 2022

Proceedings of the 59th ACM/IEEE Design Automation Conference

View full text Add to dashboard Cite

Benefiting from the search efficiency, differentiable neural architecture search (NAS) has evolved as the most dominant alternative to automatically design competitive deep neural networks (DNNs). We note that DNNs must be executed under strictly hard performance constraints in real-world scenarios, for example, the runtime latency on autonomous vehicles. However, to obtain the architecture that meets the given performance constraint, previous hardwareaware differentiable NAS methods have to repeat a plethora of search runs to manually tune the hyper-parameters by trial and error, and thus the total design cost increases proportionally. To resolve this, we introduce a lightweight hardware-aware differentiable NAS framework dubbed LightNAS, striving to find the required architecture that satisfies various performance constraints through a one-time search (i.e., you only search once). Extensive experiments are conducted to show the superiority of LightNAS over previous state-of-the-art methods. Related codes will be released at https://github.com/stepbuystep/LightNAS.

show abstract

Multi-Task Learning for Multi-Objective Evolutionary Neural Architecture Search

Cited by 21 publications

References 17 publications

Bi-Level Multiobjective Evolutionary Learning: A Case Study on Multitask Graph Neural Topology Search

Bi-Level Multiobjective Evolutionary Learning: A Case Study on Multitask Graph Neural Topology Search

Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural Architecture Search

You only search once

Contact Info

Product

Resources

About