Motivation: Automated identification of thoracic diseases from chest X-ray images (CXR) is a significant area in computer-aided diagnosis. However, most existing methods have limited ability to extract multi-scale features and accurately capture the spatial location of lesions when dealing with thoracic diseases that exhibit concurrency and large variations in lesion size. Method: Based on the above problems, we propose a multi-level residual feature fusion network (MLRFNet) for classifying thoracic diseases. Our approach can quickly capture receptive field information across different lesion sizes and enhance disease-specific features within the spatial domain on feature maps. The MLRFNet comprises two main components: a feature extractor that learns multi-scale semantic information from chest X-ray images and a multi-level residual feature classifier (MRFC) that refines disease-specific pathological features at spatial locations to reduce interference from irrelevant regions. Additionally, the ECA attention modules connect both components to enable flexible channel-wise focus on critical pathological information. Results: We evaluated the performance of MLRFNet through a series of experiments on two datasets: ChestX-Ray14 and CheXpert. Our results show that MLRFNet achieves an average AUC of 0.853 on the ChestX-Ray14 dataset and 0.904 on the CheXpert dataset. Conclusion: The results of experiments demonstrate that our proposed method works better than the current state-of-the-art baselines. Future work will focus on investigating the interdependencies among labels for thoracic diseases and techniques for model compression.INDEX TERMS Chest X-ray image classification, convolutional neural network, attention mechanism, residual feature vector, medical image processing.