A Training Method for Low Rank Convolutional Neural Networks Based on Alternating Tensor Compose-Decompose Method

Abstract

Over the past decade, deep learning-based computer vision methods have been shown to surpass previous state-of-the-art computer vision techniques in various fields, and have made great progress in various computer vision problems, including object detection, object segmentation, face recognition, etc. Nowadays, major IT companies are adding new deep-learning-based computer technologies to edge devices such as smartphones. However, since the computational cost of deep learning-based models is still high for edge devices, research is being actively carried out to compress deep learning-based models while not sacrificing high performance. Recently, many lightweight architectures have been proposed for deep learning-based models which are based on low-rank approximation. In this paper, we propose an alternating tensor compose-decompose (ATCD) method for the training of low-rank convolutional neural networks. The proposed training method can better train a compressed low-rank deep learning model than the conventional fixed-structure based training method, so that a compressed deep learning model with higher performance can be obtained in the end of the training. As a representative and exemplary model to which the proposed training method can be applied, we propose a rank-1 convolutional neural network (CNN) which has a structure alternatively containing 3-D rank-1 filters and 1-D filters in the training stage and a 1-D structure in the testing stage. After being trained, the 3-D rank-1 filters can be permanently decomposed into 1-D filters to achieve a fast inference in the test time. The reason that the 1-D filters are not being trained directly in 1-D form in the training stage is that the training of the 3-D rank-1 filters is easier due to the better gradient flow, which makes the training possible even in the case when the fixed structured network with fixed consecutive 1-D filters cannot be trained at all. We also show that the same training method can be applied to the well-known MobileNet architecture so that better parameters can be obtained than with the conventional fixed-structure training method. Furthermore, we show that the 1-D filters in a ResNet like structure can also be trained with the proposed method, which shows the fact that the proposed method can be applied to various structures of networks.

Efficiency of the project method in the development of professional competencies in future teachers

“…The pedagogical principle "from simple to complex", the movement from knowledge and skills to competencies should be reflected in the organisational structure of training, which is proposed to include three levels: higher level -project training with the development and implementation of professional projects evaluated by employers, intermediate level -development of training and proto-professional projects, with their evaluation by top-level teachers and at Ukrainian student rating competitions, initial level -traditional disciplinary training, with the evaluation and formation of "orders" for the evaluated product by middle-level teachers. This idea of implementing the project method is confirmed in the works of Lee et al (2021), Shuhailo and Derkach (2021).…”

“…Therefore, it is advisable to organise learning and research work of students, the use of business, role-playing, simulation games in the creative independent work of students (Lee et al, 2021).…”

Project Method Efficiency for the Teachers’ Professional Activities

“…The researchers note that the achievement of high efficiency of the project method is impossible without full cooperation of the educational process participants. Lee et al (2021) emphasize the important role of teachers. According to researchers, teachers who are ready for education innovations play the main role in the project method implementation.…”

New convolutional neural network models for efficient object recognition with humanoid robots

“…Recent works on effective deep learning have focused on model compression and reducing the computational operations in DNNs (Chen et al, 2015;Denil et al, 2013;Han et al, 2015;Idelbayev & Carreira-Perpinán, 2021;Lee et al, 2021;Rastegari et al, 2016). Chen et al (2015) presented a new network architecture called HashedNets that takes the advantage of inherent redundancy.…”

“…Jaderberg et al (2014) used a cross-channel or filter redundancy to generate low-rank filters for speeding up pre-trained CNNs with minimal loss of accuracy. In Lee et al (2021); Idelbayev and Carreira-Perpinán (2021), a tensor compose-decompose approximation and an approximation combining low-rank decompositions, using different matrix shapes, were proposed to obtain the compressed DNNs with high performance.…”