Towards Less Constrained Macro-Neural Architecture Search

Abstract

Networks found with Neural Architecture Search (NAS) achieve state-of-the-art performance in a variety of tasks, out-performing human-designed networks. However, most NAS methods heavily rely on human-defined assumptions that constrain the search: architecture's outer-skeletons, number of layers, parameter heuristics and search spaces. Additionally, common search spaces consist of repeatable modules (cells) instead of fully exploring the architecture's search space by designing entire architectures (macrosearch). Imposing such constraints requires deep human expertise and restricts the search to pre-defined settings. In this paper, we propose LCMNAS, a method that pushes NAS to less constrained search spaces by performing macrosearch without relying on pre-defined heuristics or bounded search spaces. LCMNAS introduces three components for the NAS pipeline: i) a method that leverages information about well-known architectures to autonomously generate complex search spaces based on Weighted Directed Graphs with hidden properties, ii) a evolutionary search strategy that generates complete architectures from scratch, and iii) a mixed-performance estimation approach that combines information about architectures at initialization stage and lower fidelity estimates to infer their trainability and capacity to model complex functions. We present experiments showing that LCMNAS generates state-of-the-art architectures from scratch with minimal GPU computation. We study the importance of different NAS components on a macro-search setting. Code for reproducibility is public at https://github.com/VascoLopes/LCMNAS.

Systematic Review on Neural Architecture Search

“…• Population generation: some architectures should be chosen as an initial population; • Performance evaluation: the algorithm will evaluate the population's performance based on some metrics such as accuracy, complexity, and resource requirement; • Population evolution: some models will be chosen based on selection techniques, such as tournament selection [17] or fitness proportionate selection [58], to evolve and make the next generations; • Termination criteria: this evolutionary procedure continues until a termination criterion, such as a predefined error or a maximum number of generations, is met.…”