Spectral-based convolutional neural network without multiple spatial-frequency domain switchings

A Low-complexity Complex-valued Activation Function for Fast and Accurate Spectral Domain Convolutional Neural Network

“…Thus, this approach requires multiple spatial-spectral domain switching. These domain transformations are computationally intensive, and hence negate some of the gain in computational complexity achieved with spectral domain CNN [9,12,14]. In some of these approaches spectral pooling is performed to downsample spectral domain feature maps, which is considered equivalent to max pooling in spatial domain [11].…”

“…Figure 1 gives the functional block diagram that illustrates the realization of LeNet-5 CNN modeled using the abovementioned conventional approach in spectral domain CNN. Since full-connection and softmax layers are not very computationally intensive, one can keep these layers in spatial domain [14]. That is why after the last layer of the feature-learning segment, the feature maps are transformed to spatial domain.…”

“…The square function is proposed for the spatial function, which is equivalent to auto-convolution in Fourier space (Fourier-space input convolved with itself) and hence very computationally intensive. Ayat et al in [14] proposed a spectral version of ReLU, the most commonly used activation function in CNNs. This activation function, called SReLU, approximates ReLU with a quadratic function using the Taylor series and computes its Fourier equivalent.…”

“…Pratt et al in [21] has also developed another spectral domain CNN model without multiple domain transformations, but do not specify the type of activation function they use or whether they have used any activation function at all. More recently, Ayat et al in [14] proposed another spectral model for CNN without multiple domain transformations. They apply their SReLU spectral activation function and implement a fused version of convolution in spectral domain where pooling operation is merged with convolution.…”

“…First, proposed activation function does not require computationally expensive domain switchings, as the activation function works in complex domain. Second, it is computationally much lighter than state-of-the-art spectral activation functions such as SReLU [14].…”