Nonlinear Hyperparameter Optimization of a Neural Network in Image Processing for Micromachines

Abstract

Deep neural networks are widely used in the field of image processing for micromachines, such as in 3D shape detection in microelectronic high-speed dispensing and object detection in microrobots. It is already known that hyperparameters and their interactions impact neural network model performance. Taking advantage of the mathematical correlations between hyperparameters and the corresponding deep learning model to adjust hyperparameters intelligently is the key to obtaining an optimal solution from a deep neural network model. Leveraging these correlations is also significant for unlocking the “black box” of deep learning by revealing the mechanism of its mathematical principle. However, there is no complete system for studying the combination of mathematical derivation and experimental verification methods to quantify the impacts of hyperparameters on the performances of deep learning models. Therefore, in this paper, the authors analyzed the mathematical relationships among four hyperparameters: the learning rate, batch size, dropout rate, and convolution kernel size. A generalized multiparameter mathematical correlation model was also established, which showed that the interaction between these hyperparameters played an important role in the neural network’s performance. Different experiments were verified by running convolutional neural network algorithms to validate the proposal on the MNIST dataset. Notably, this research can help establish a universal multiparameter mathematical correlation model to guide the deep learning parameter adjustment process.

A TL_FLAT Model for Chinese Text Datasets of UAV Power Systems: Optimization and Performance

“…Te head_dim is the dimensionality of the multihead attention-based hidden layer of the model, which afects the performance and computational power of the model when the number of heads is a fxed value (generally 8). Additionally, we use dropout in the multihead attention layer to reduce the multihead attention weight to reduce overftting and improve the running speed of the model, and we set the parameter of the dropout function to 0.3-0.5 with reference to the literature [57]. Terefore, to obtain better entity recognition performance from the FLAT model, we conducted exploratory experiments on the optimization algorithm, learning rate (lr), number of model encoding layers, and head_dim with the CLUENER2020 dataset.…”

“…Unlike the original FLAT model, we fnd the optimal model by experimenting with optimization models such as SGD and Adam and diferent parameters (layers, lr, head_dim); then, the experimentally derived optimal model is compared with the more classic NER method to verify the superiority of the resulting model. Additionally, the parameter of the dropout function is set with reference to the literature [57] as 0.3-0.5, and the multihead attention matrix is changed to a sparse matrix by using dropout in the multihead attention layer to improve the running speed of our model. Other parameter settings are derived from the literature [52].…”

An Automated Hyperparameter Tuning Recurrent Neural Network Model for Fruit Classification

“…It provides an overview of various CNN approaches utilized for image classification, segmentation, and styling. Next, in [19], the mathematical relationship between four hyperparameters, namely learning rate, batch size, dropout rate, and convolution kernel size were investigated in detail. A generalized multi-parameter mathematical correlation approach was derived, showing that the hyperparameters play a vital part in the efficiency of the NN models.…”

AAL and Internet of Medical Things for Monitoring Type-2 Diabetic Patients

“…A dense-layer neuron conducts matrix-vector multiplication over the input of each neuron in the previous layer. As indicated in the Equation, the usual formula for matrix-vector multiplication is as follows (7) [ 37 ]: …”