Traditional pottery identification methods are time consuming and costly. In order to cater to more pottery industry needs, we propose a Mask R-CNN-based pottery identification method to build an automatic pottery identification system. We first improve the loss function of Mask R-CNN by using generalized intersection over union loss function, through the pattern of migration learning, to compensate for the disadvantage of a small collective amount of pottery data. For different types of pottery, we use the mask algorithm to enhance the features of the outer contour of the pottery. In addition, we use the minimum external matrix algorithm to accurately extract the outer contour bit pose features of pottery to improve the model’s accuracy in recognizing the outer contour of pottery. To meet the testing conditions of pottery, with the support of potters and archaeologists, we make our pottery data set according to pottery categories. The experimental results prove that our method performs best in the comprehensive recognition accuracy of pottery, with the recognition accuracy above 90%. The recognition accuracy is also the best in pottery color decoration and grain decoration, and the grain recognition accuracy stays above 87%, which is better than other pottery recognition methods.