Reliably and effectively detecting and classifying leather surface defects is of great importance to tanneries and industries that use leather as a major raw material such as leather footwear and handbag manufacturers. This paper presents a detailed and methodical review of the leather surface defects, their effects on leather quality grading and automated visual inspection methods for leather defect inspection. A detailed review of inspection methods based on leather defect detection using image analysis methods is presented, which are usually classified as heuristic or basic machine learning based methods. Due to the recent success of deep learning methods in various related fields, various architectures of deep learning are discussed that are tailored to image classification, detection, and segmentation. In general, visual inspection applications, where recent CNN architectures are classified, compared, and a detailed review is subsequently presented on the role of deep learning methods in leather defect detection. Finally, research guidelines are presented to fellow researchers regarding data augmentation, leather quality quantification, and simultaneous defect inspection methods, which need to be investigated in the future to make progress in this crucial area of research.