Joint design and compression of convolutional neural networks as a Bi-level optimization problem

Abstract

Over the last decade, deep neural networks have shown great success in the fields of machine learning and computer vision. Currently, the CNN (convolutional neural network) is one of the most successful networks, having been applied in a wide variety of application domains, including pattern recognition, medical diagnosis and signal processing. Despite CNNs’ impressive performance, their architectural design remains a significant challenge for researchers and practitioners. The problem of selecting hyperparameters is extremely important for these networks. The reason for this is that the search space grows exponentially in size as the number of layers increases. In fact, all existing classical and evolutionary pruning methods take as input an already pre-trained or designed architecture. None of them take pruning into account during the design process. However, to evaluate the quality and possible compactness of any generated architecture, filter pruning should be applied before the communication with the data set to compute the classification error. For instance, a medium-quality architecture in terms of classification could become a very light and accurate architecture after pruning, and vice versa. Many cases are possible, and the number of possibilities is huge. This motivated us to frame the whole process as a bi-level optimization problem where: (1) architecture generation is done at the upper level (with minimum NB and NNB) while (2) its filter pruning optimization is done at the lower level. Motivated by evolutionary algorithms’ (EAs) success in bi-level optimization, we use the newly suggested co-evolutionary migration-based algorithm (CEMBA) as a search engine in this research to address our bi-level architectural optimization problem. The performance of our suggested technique, called Bi-CNN-D-C (Bi-level convolution neural network design and compression), is evaluated using the widely used benchmark data sets for image classification, called CIFAR-10, CIFAR-100 and ImageNet. Our proposed approach is validated by means of a set of comparative experiments with respect to relevant state-of-the-art architectures.

Systematic Review on Neural Architecture Search

“…Sometimes, the NAS algorithms tend to find complex and large-scale architectures with high accuracy which are not efficient for industrial or real-world applications. To solve this problem, Louati et al [103] used EA optimizer together with a pruning method to make the network fit the hardware requirements like suitable memory size for health-care applications. This method works in two-step sequences: the low-level pruning action and a high-level design process, by implementing a co-evolutionary migration-based algorithm.…”

Advancing Sustainable COVID-19 Diagnosis: Integrating Artificial Intelligence with Bioinformatics in Chest X-ray Analysis

“…In the realm of image classification using deep neural networks, the most commonly employed performance metrics, as indicated by the literature [43,44], are accuracy (Acc), specificity, and sensitivity. Accuracy (Acc), denoted by Equation (1), is defined as the ratio of true positives (TP) and true negatives (TN) to the total number of cases (NE).…”

Two-Phase Evolutionary Convolutional Neural Network Architecture Search for Medical Image Classification

“…The efficiency and effectiveness of the proposed TPEvo-CNN is compared with the state-of-the-art CNN models regarding classification metrics such as accuracy, precision, recall and F1-score concerning four COVID-19 datasets categorized based on image types and a number of classes. We have considered the compared CNN models such as COVID-Net [44], MobileNet [45], DarkCovidNet [46], CNN-SA [47], CoroNet [48], GSA-DenseNet121-COVID-19 [49], DRENet [50], Deep-chest [51], Bi-CNN-D-C [52], CNGOD [53] because these models have been experimented on the same category of COVID-19 datasets as carried in our work. Notably, the reproducibility of the developed models for the purpose of comparison may only sometimes be feasible due to the unavailability of the code implementation, computational resources, and the requirements of high execution times as recommended in the literature [12], [13].…”