Electrical overhead line towers are painted to protect their metal surfaces from direct interaction with the environment. Subsequently, paint is applied to refurbish exposed towers. On a vast network, it is difficult to identify which line segments or towers require refurbishment. Industry practice involves taking aerial images of towers and classifying the level of paint defects, albeit manually. This process is labour-intensive and subjective. In this paper, we propose a prototype system based on deep learning to automatically identify towers at risk due to paint deterioration. We use a representative tower inspection data set from the industry with 343k images of 6,333 towers for development and evaluation. Each tower is classified as being within normal operating conditions or at high risk. This is achieved by aggregating class predictions from each of the multiple images of the tower. Supervised learning used only tower-level condition labels; no annotation of individual images or image regions was used. Prototype systems based on EfficientNets achieved a test area under the ROC curve of 0.97. A true positive rate of 0.98 was obtained for a corresponding false positive rate of 0.14. Thus, we demonstrate that towers at risk from significant paintwork deterioration can be identified effectively, and that tower-level labels are adequate for training, eliminating the need for the costly annotation of sub-tower parts.