Defects detection with Electroluminescence (EL) image for photovoltaic (PV) module has become a standard test procedure during the process of production, installation, and operation of solar modules. There are some typical defects types, such as crack, finger interruption, that can be recognized with high accuracy. However, due to the complexity of EL images and the limitation of the dataset, it is hard to label all types of defects during the inspection process. The unknown or unlabeled create significant difficulties in the practical application of the automatic defects detection technique. To address the problem, we proposed an evolutionary algorithm combined with traditional image processing technology, deep learning, transfer learning, and deep clustering, which can recognize the unknown or unlabeled in the original dataset defects automatically along with the increasing of the dataset size. Specifically, we first propose a deep learning-based features extractor and defects classifier. Then, the unlabeled defects can be classified by the deep clustering algorithm and stored separately to update the original database without human intervention. When the number of unknown images reaches the preset values, transfer learning is introduced to train the classifier with the updated database. The fine-tuned model can detect new defects with high accuracy. Finally, numerical results confirm that the proposed solution can carry out efficient and accurate defect detection automatically using electroluminescence images.