Gradient Descent Effects on Differential Neural Architecture Search: A Survey

Santra, Santanu; Hsieh, Jun-Wei; Lin, Chia-Li

doi:10.1109/access.2021.3090918

Cited by 29 publications

(13 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Liang et al [ 34 ] observed performance collapse as the number of epochs of DARTS increased, and found that this was a result of overfitting due to the number of skip-connections that increased with epochs. They solved the performance collapse problem through DARTS+ [ 34 ], which applied the ‘early stopping’ technique, based on the research results that important connections and significant changes were determined in the early phase of training [ 49 , 50 , 51 ]. Chu et al proposed Fair-DARTS [ 35 ], which applied independence of each operation’s architectural weight to solve the performance collapse problem caused by skip-connection, and eliminated the unfair advantage.…”

Section: Neural Architecture Searchmentioning

confidence: 99%

Neural Architecture Search Survey: A Computer Vision Perspective

Kang

Kang²,

Kim

et al. 2023

Sensors

View full text Add to dashboard Cite

In recent years, deep learning (DL) has been widely studied using various methods across the globe, especially with respect to training methods and network structures, proving highly effective in a wide range of tasks and applications, including image, speech, and text recognition. One important aspect of this advancement is involved in the effort of designing and upgrading neural architectures, which has been consistently attempted thus far. However, designing such architectures requires the combined knowledge and know-how of experts from each relevant discipline and a series of trial-and-error steps. In this light, automated neural architecture search (NAS) methods are increasingly at the center of attention; this paper aimed at summarizing the basic concepts of NAS while providing an overview of recent studies on the applications of NAS. It is worth noting that most previous survey studies on NAS have been focused on perspectives of hardware or search strategies. To the best knowledge of the present authors, this study is the first to look at NAS from a computer vision perspective. In the present study, computer vision areas were categorized by task, and recent trends found in each study on NAS were analyzed in detail.

show abstract

Section: Neural Architecture Searchmentioning

confidence: 99%

Neural Architecture Search Survey: A Computer Vision Perspective

Kang

Kang²,

Kim

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…According to the chain rule, the backpropagation algorithm uses the error between the expected and the actual output as the backpropagation, and uses the gradient descent method to adjust the network parameters to promote the error to develop in the direction of smaller [ 31 , 32 , 33 ]. The principle of the gradient descent algorithm is to find the extreme point of the objective function

, which is the point where the derivative

.…”

Section: Fpga Design Of Bp Neural Network Pid Algorithmmentioning

confidence: 99%

A Design of FPGA-Based Neural Network PID Controller for Motion Control System

Wang

Jiang

et al. 2022

Sensors

View full text Add to dashboard Cite

In the actual industrial production process, the method of adaptively tuning proportional–integral–derivative (PID) parameters online by neural network can adapt to different characteristics of different controlled objects better than the controller with PID. However, the commonly used microcontroller unit (MCU) cannot meet the application scenarios of real time and high reliability. Therefore, in this paper, a closed-loop motion control system based on BP neural network (BPNN) PID controller by using a Xilinx field programmable gate array (FPGA) solution is proposed. In the design of the controller, it is divided into several sub-modules according to the modular design idea. The forward propagation module is used to complete the forward propagation operation from the input layer to the output layer. The PID module implements the mapping of PID arithmetic to register transfer level (RTL) and is responsible for completing the output of control amount. The main state machine module generates enable signals that control the sequential execution of each sub-module. The error backpropagation and weight update module completes the update of the weights of each layer of the network. The peripheral modules of the control system are divided into two main parts. The speed measurement module completes the acquisition of the output pulse signal of the encoder and the measurement of the motor speed. The pulse width modulation (PWM) signal generation module generates PWM waves with different duty cycles to control the rotation speed of the motor. A co-simulation of Modelsim and Simulink is used to simulate and verify the system, and a test analysis is also performed on the development platform. The results show that the proposed system can realize the self-tuning of PID control parameters, and also has the characteristics of reliable performance, high real-time performance, and strong anti-interference. Compared with MCU, the convergence speed is far more than three orders of magnitude, which proves its superiority.

show abstract

“…The search strategy specifies the algorithm used to search for the optimal architecture. These algorithms include: random search [27], Bayesian optimization [28], evolutionary algorithms [26], reinforcement learning [29], and gradient-based algorithms [30]. Among them, Google's reinforcement learning search method was an earlier exploration in 2017.…”

Section: Nas Problem Black Box Modelingmentioning

confidence: 99%

Quantum Dynamic Optimization Algorithm for Neural Architecture Search on Image Classification

Jin

Zhang²,

et al. 2022

Electronics

View full text Add to dashboard Cite

Deep neural networks have proven to be effective in solving computer vision and natural language processing problems. To fully leverage its power, manually designed network templates, i.e., Residual Networks, are introduced to deal with various vision and natural language tasks. These hand-crafted neural networks rely on a large number of parameters, which are both data-dependent and laborious. On the other hand, architectures suitable for specific tasks have also grown exponentially with their size and topology, which prohibits brute force search. To address these challenges, this paper proposes a quantum dynamic optimization algorithm to find the optimal structure for a candidate network using Quantum Dynamic Neural Architecture Search (QDNAS). Specifically, the proposed quantum dynamics optimization algorithm is used to search for meaningful architectures for vision tasks and dedicated rules to express and explore the search space. The proposed quantum dynamics optimization algorithm treats the iterative evolution process of the optimization over time as a quantum dynamic process. The tunneling effect and potential barrier estimation in quantum mechanics can effectively promote the evolution of the optimization algorithm to the global optimum. Extensive experiments on four benchmarks demonstrate the effectiveness of QDNAS, which is consistently better than all baseline methods in image classification tasks. Furthermore, an in-depth analysis is conducted on the searchable networks that provide inspiration for the design of other image classification networks.

show abstract

Gradient Descent Effects on Differential Neural Architecture Search: A Survey

Cited by 29 publications

References 31 publications

Neural Architecture Search Survey: A Computer Vision Perspective

Neural Architecture Search Survey: A Computer Vision Perspective

A Design of FPGA-Based Neural Network PID Controller for Motion Control System

Quantum Dynamic Optimization Algorithm for Neural Architecture Search on Image Classification

Contact Info

Product

Resources

About