Photovoltaic (PV) cell defect detection has become a prominent problem in the development of the PV industry; however, the entire industry lacks effective technical means. In this paper, we propose a deep-learning-based defect detection method for photovoltaic cells, which addresses two technical challenges: (1) to propose a method for data enhancement and category weight assignment, which effectively mitigates the impact of the problem of scant data and data imbalance on model performance; (2) to propose a feature fusion method based on ResNet152–Xception. A coordinate attention (CA) mechanism is incorporated into the feature map to enhance the feature extraction capability of the existing model. The proposed model was conducted on two global publicly available PV-defective electroluminescence (EL) image datasets, and using CNN, Vgg16, MobileNetV2, InceptionV3, DenseNet121, ResNet152, Xception and InceptionResNetV2 as comparative benchmarks, it was evaluated that several metrics were significantly improved. In addition, the accuracy reached 96.17% in the binary classification task of identifying the presence or absence of defects and 92.13% in the multiclassification task of identifying different defect types. The numerical experimental results show that the proposed deep-learning-based defect detection method for PV cells can automatically perform efficient and accurate defect detection using EL images.