Solar energy can be a clean and renewable alternative to traditional fuels, which enables its wide application in our life and the industry. However, some defects inevitably occur in the solar cells during production, transportation, and installation, which will reduce the power generation efficiency. In this paper, we propose a ResNet-based micro-crack detection method to detect the micro-cracks on polycrystalline solar cells. Specifically, a novel feature fusion model is introduced to aggregate the low-level features and deep semantically strong features by self-attention mechanism to obtain accurate geometry information. This method boosts the detection accuracy to 99.11%, which significantly surpasses other counterparts, e.g., some state-of-the-art deep neural networks, by a large margin. Since it is difficult for other methods to precisely detect other defect types apart from micro-cracks, we further propose a transfer learning method based on MK-MMD to guide the training process of defect detector with another pre-trained micro-crack detector. With the help of transfer learning, the accuracy of solar cell defect detection increases by 11.6%.