Pavement deterioration and abnormal climate induced by global warming lead to a constant rise in the number of potholes. Accordingly, the loss cost for maintenance and accidents also increases. Therefore, it is necessary to develop a method of classifying pavement potholes and detecting their locations. This study proposes the pothole region extraction based on similarity evaluation scale classification using image processing. The proposed technique sets up a classification threshold appropriately by considering the structure, brightness, and other factors of the grayscale-converted image through SSIM (Structural Similarity Index Measure). It binarizes porthole images classified according to the threshold, and then extracts pothole regions through the threshold based segmentation. A conventional image classification method utilizes the rules found in objects or the label selected by a user. The proposed method can take into account detailed factors by comparing image similarity in the unit of pixel. According to the performance evaluation, the proposed classification method’s F1-score is 0.83, and its accuracy of pothole region extraction is 0.851. Therefore, with the proposed technique, it is possible to make classification in consideration of similarity between images. In addition, the proposed method makes it possible to detect the regions similar to actual potholes accurately.