How to Tell Ancient Signs Apart? Recognizing and Visualizing Maya Glyphs with CNNs

Abstract

Thanks to the digital preservation of cultural heritage material, multimedia tools, e.g. based on automatic visual processing, enable to considerably ease the work of scholars in the humanities and help them to perform quantitative analysis of their data. In this context, this paper assesses three different Convolutional Neural Network (CNN) architectures along with three learning approaches to train them for hieroglyph classification, which is a very challenging task due to the limited availability of segmented ancient Maya glyphs. More precisely, the first approach, the baseline, relies on pretrained networks as feature extractor. The second one investigates a transfer learning method by fine-tuning a pretrained network for our glyph classification task. The third approach considers directly training networks from scratch with our glyph data. The merits of three different network architectures are compared: a generic sequential model (i.e. LeNet), a sketch-specific sequential network (i.e. Sketch-a-Net), and the recent Residual Networks. The sketch-specific model trained from scratch outperforms other models and training strategies. Even for a challenging 150-class classification task, this model achieves 70.3% average accuracy and proves itself promising in case of small amount of cultural heritage shape data. Furthermore, we visualize the discriminative parts of glyphs with the recent Grad-CAM method, and demonstrate that the discriminative parts learned by the model agrees in general with the expert annotation of the glyph specificity (diagnostic features). Finally, as a step towards systematic evaluation of these visualizations, we conduct a perceptual crowdsourcing study. Specifically, we analyze the interpretability of the representations from Sketch-a-Net and ResNet-50. Overall, our paper takes two important steps towards providing tools to scholars in the digital humanities: increased performance for automation, and improved interpretability of algorithms. CCS Concepts: • Computing methodologies → Shape representations; Neural networks; Object identification; • Applied computing → Arts and humanities;

Digital Restoration of Cultural Heritage With Data-Driven Computing: A Survey

“…DL and ML are powerful tools to automate this effort rather than human intervention. [66] proposed a method to crowdsource the data in the first place to collect the dataset. Firstly, they take this dataset into a step of preprocessing for the oversampling task.…”

“…Training any network for hieroglyph classification is challenging since segmented Maya glyphs are not available in plenty. Can et al [66] make use of three different CNN architectures to train them for classifying hieroglyphs? The first approach uses a pre-trained CNN as a feature extractor.…”

“…We explore some important datasets [63], [64], [65] to accelerate the effect of DL. The development surrogate in the form of ancient script classification such as Mayan [66], and Egyptian [67].…”

A Few-shot Learning Approach for Historical Ciphered Manuscript Recognition

“…With the recent advances in deep learning [3], HTR systems can reach a very high performance [4], especially for modern documents with legible handwriting styles, known language, vocabulary and syntax. Contrary, the recognition of historical manuscripts is still challenging due to paper degradation, old vocabularies, uncommon handwriting styles, etc [5], [6], [7]. To partially cope with these difficulties, context information is often incorporated through specific language models, dictionaries, etc.…”

Using Deep Learning for the Image Recognition of Motifs on the Center of Sukhothai Ceramics

“…The style of painting, created in Dunhuang, China, also strongly related to the era. Later on, Can et al [13] studied three CNN architectures: Sketch-a-Net, VGG-16, and ResNet-50. Their study compared the efficiency of Mayan hieroglyphics classification from the Maya Codice dataset [14].…”