Automation improves the quality of fruits through quick and accurate detection of pest and disease infections thus contributing to the country's economic growth and productivity. Although humans can identify the fruit damage caused by pests and diseases, methods used are inconsistent, time-consuming, and variable. The surface features of fruits typically observed by consumers who seek their health benefits, affect their market value. The issue of pest and disease infections further deteriorates fruits' quality, becoming a mounting stressor on farmers as they affect the potential income that could have been realised from production, processing and export. This article reviews various studies on detecting and classifying damages in fruits. Specifically, we review articles where state-of-the-art approaches under segmentation, image processing, machine learning, and deep learning have proved effective in developing automated systems that address hurdles associated with manual methods of assessing damage using visual experiences. This survey reviews 32 Journal and Conference articles spanning 13 years obtained electronically through Google Scholar, Scopus, IEEE, ScienceDirect, and general internet searches. This survey further presents a detailed discussion of related studies done in the past while emphasizing their strengths and limitations and presenting future research directions. It also reveals that much as the use of automated detection and classification of fruit damage has yielded promising results in the horticulture industry, more research is still needed with systems required to fully automate the detection and classification processes, especially those that are mobile phone-based towards addressing occlusion challenges.INDEX TERMS fruit damage detection, classification, deep learning, image analysis and segmentation.