Deep convolutional networks have better smoke recognition performance. However, a lightweight network model and high recognition accuracy cannot be balanced when deployed on hardware with limited computing resources such as edge computing. Based on this background, we propose a novel smoke recognition network that combines convolutional networks (CNN) and self-attention. The core ideas of this framework are as follows: (1) Combine the depthwise convolution and asymmetric convolution of large convolution kernels to construct a lightweight CNN model, and realize multiscale extraction of feature information with slight model complexity. (2) Combined with the self-attention in transformer, a skip-connection branch is designed, which improves the feature extraction capability of the backbone network through parallel processing and fusion of feature map information. (3) Fusion multicomponent discrete cosine transform (DCT) is used to compress channel information and expand the ability of global average pooling (GAP) to aggregate feature maps. The proposed DCT-GAP improves the accuracy of the network without adding additional computational costs. Experimental results show that the proposed CSANet achieves an average accuracy of over 98.3% with 238 M FLOPs and 5.8 M parameters on the homemade smoke dataset, outperforming state-of-the-art competitors.