Defective PV panels reduce the efficiency of the whole PV string, causing loss of investment by decreasing its efficiency and lifetime. In this study, firstly, an isolated convolution neural model (ICNM) was prepared from scratch to classify the infrared images of PV panels based on their health, i.e., healthy, hotspot, and faulty. The ICNM occupies the least memory, and it also has the simplest architecture, lowest execution time, and an accuracy of 96% compared to transfer learned pre-trained ShuffleNet, GoogleNet, and SqueezeNet models. Afterward, ICNM, based on its advantages, is reused through transfer learning to classify the defects of PV panels into five classes, i.e., bird drop, single, patchwork, horizontally aligned string, and block with 97.62% testing accuracy. This proposed approach can identify and classify the PV panels based on their health and defects faster with high accuracy and occupies the least amount of the system’s memory, resulting in savings in the PV investment.