A New Pointwise Convolution in Deep Neural Networks Through Extremely Fast and Non Parametric Transforms

Abstract

Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms can serve as a powerful feature extractor in channel dimension without any learnable parameters in deep neural networks. This paper firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms can significantly reduce the computational complexity of neural networks without accuracy degradation on various classification tasks and even on face detection task. Our comprehensive experiments show that the proposed DWHT-based model gained 1.49% accuracy increase with 79.4% reduced parameters and 49.4% reduced FLOPs compared with its baseline model on the CIFAR 100 dataset while achieving comparable accuracy under the condition that 81.4% of parameters and 49.4% of FLOPs reduced on SVHN dataset. Additionally, our DWHT-based model showed comparable accuracy with 89.2% reduced parameters and 26.5% reduced FLOPs compared to the baseline models on WIDER FACE and FDDB datasets. INDEX TERMS Efficient deep neural network architecture, pointwise convolution, discrete Walsh-Hadamard transform, discrete cosine transform

Accelerating the Fast Hadamard Single-Pixel Imaging (FHSI) on GPUs: Techniques and Optimizations

“…Comparative studies of different SPI reconstructed methods Moreover, HSI under GPU acceleration has shown that it depends on the specific application requirements, including the nature of the data, computational environment, and Deep Learning [41,42]. With its simpler arithmetic operations, the HSI often exhibits advantages in scenarios where the computation can be entirely kept within the real domain.…”

Crop Disease Identification Method Based on Convolution Neural Network

“…For training samples, it is not easy to get the required data set, which makes the network fall into the local optimal solution to find the best parameter set. In the process of practical application, it is usually necessary to determine the objective function value after multiple iterations [17][18]. When we do not want this parameter to achieve our desired purpose, we cannot use this method.…”