Learning strides in convolutional neural networks

Abstract

Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations. This provides some shift-invariance while reducing the computational complexity of the whole architecture. A critical hyperparameter of such layers is their stride: the integer factor of downsampling. As strides are not differentiable, finding the best configuration either requires cross-validation or discrete optimization (e.g. architecture search), which rapidly become prohibitive as the search space grows exponentially with the number of downsampling layers. Hence, exploring this search space by gradient descent would allow finding better configurations at a lower computational cost. This work introduces \ours{}, the first downsampling layer with learnable strides. Our layer learns the size of a cropping mask in the Fourier domain, that effectively performs resizing in a differentiable way. Experiments on audio and image classification show the generality and effectiveness of our solution: we use \ours{} as a drop-in replacement to standard downsampling layers and outperform them. In particular, we show that introducing our layer into a \resnet{} architecture allows keeping consistent high performance on CIFAR10, CIFAR100 and ImageNet even when training starts from poor random stride configurations. Moreover, formulating strides as learnable variables allows us to introduce a regularization term that controls the computational complexity of the architecture. We show how this regularization allows trading off accuracy for efficiency on ImageNet.

COVID-19 diagnosis via chest X-ray image classification based on multiscale class residual attention

“…With the emerge of deep neural networks (DNNs) [ [9] , [10] , [11] , [12] , [13] , [14] ], especially convolutional neural networks (CNNs), they leverage multi-level layer neural networks for representational learning and are widely used for image classification [ 15 , 16 ], object detection [ 17 , 18 ] and semantic segmentation [ 19 ]. Naturally, DNNs are very good at detecting COVID-19 [ [20] , [21] , [22] , [23] , [24] , [25] ].…”

Learning Temporal Resolution in Spectrogram for Audio Classification

“…All the datasets are resampled at a sampling rate of 16 kHz. Following the evaluation protocol in the previous works (Zeghidour et al 2021;Riad et al 2021;Kong et al 2020;Gong, Chung, and Glass 2021b), we report the mean average precision (mAP) as the main evaluation metric on AudioSet and FSD50K, and report classification accuracy (ACC) on other datasets. In all experiments, we use the same architecture as used by Gong, Chung, and Glass (2021b), which is an EfficientNet-B2 (Tan and Le 2019) with four attention heads (13.6 M parameters).…”

“…Learning Feature Resolution. One recent work on learning feature resolution for audio classification is Diff-Stride (Riad et al 2021), which learns stride in convolutional neural network (CNN) in a differentiable way and outperforms previous methods using fixed stride settings. By comparison, DiffStride needs to be applied in each CNN layer and can only learn a single fixed stride setting, while DiffRes is a one-layer lightweight algorithm and can personalize the best temporal resolution for each audio during inference.…”

Understanding of Convolutional Neural Network (CNN): A Review

“…Kernel size is the kernel filter's size or the sliding kernel [22]. Stride length is the number of kernels that slide before making product points and creating output pixels [23]. Padding is the size of the 0-th frame set up around the input feature map [24].…”