Ship-radiated noise classification is critical in ocean acoustics. Recently, the feature extraction method combined with time–frequency spectrograms and convolutional neural networks (CNNs) has effectively described the differences between various underwater targets. However, many existing CNNs are challenging to apply to embedded devices because of their high computational costs. This paper introduces a lightweight network based on multi-scale asymmetric CNNs with an attention mechanism (MA-CNN-A) for ship-radiated noise classification. Specifically, according to the multi-resolution analysis relying on the relationship between multi-scale convolution kernels and feature maps, MA-CNN-A can autonomously extract more fine-grained multi-scale features from the time–frequency domain. Meanwhile, the MA-CNN-A maintains its light weight by employing asymmetric convolutions to balance accuracy and efficiency. The number of parameters introduced by the attention mechanism only accounts for 0.02‰ of the model parameters. Experiments on the DeepShip dataset demonstrate that the MA-CNN-A outperforms some state-of-the-art networks with a recognition accuracy of 98.2% and significantly decreases the parameters. Compared with the CNN based on three-scale square convolutions, our method has a 68.1% reduction in parameters with improved recognition accuracy. The results of ablation explorations prove that the improvements benefit from asymmetric convolution, multi-scale block, and attention mechanism. Additionally, MA-CNN-A shows a robust performance against various interferences.