Objective COVID‐19 is ravaging the world, but traditional reverse transcription‐polymerase reaction (RT‐PCR) tests are time‐consuming and have a high false‐negative rate and lack of medical equipment. Therefore, lung imaging screening methods are proposed to diagnose COVID‐19 due to its fast test speed. Currently, the commonly used convolutional neural network (CNN) model requires a large number of datasets, and the accuracy of the basic capsule network for multiple classification is limital. For this reason, this paper proposes a novel model based on CNN and CapsNet. Methods The proposed model integrates CNN and CapsNet. And attention mechanism module and multi‐branch lightweight module are applied to enhance performance. Use the contrast adaptive histogram equalization (CLAHE) algorithm to preprocess the image to enhance image contrast. The preprocessed images are input into the network for training, and ReLU was used as the activation function to adjust the parameters to achieve the optimal. Result The test dataset includes 1200 X‐ray images (400 COVID‐19, 400 viral pneumonia, and 400 normal), and we replace CNN of VGG16, InceptionV3, Xception, Inception‐Resnet‐v2, ResNet50, DenseNet121, and MoblieNetV2 and integrate with CapsNet. Compared with CapsNet, this network improves 6.96%, 7.83%, 9.37%, 10.47%, and 10.38% in accuracy, area under the curve (AUC), recall, and F1 scores, respectively. In the binary classification experiment, compared with CapsNet, the accuracy, AUC, accuracy, recall rate, and F1 score were increased by 5.33%, 5.34%, 2.88%, 8.00%, and 5.56%, respectively. Conclusion The proposed embedded the advantages of traditional convolutional neural network and capsule network and has a good classification effect on small COVID‐19 X‐ray image dataset.