Handwriting Styles: Benchmarks and Evaluation Metrics

Abstract

Evaluating the style of handwriting generation is a challenging problem, since it is not well defined. It is a key component in order to develop in developing systems with more personalized experiences with humans. In this paper, we propose baseline benchmarks, in order to set anchors to estimate the relative quality of different handwriting style methods. This will be done using deep learning techniques, which have shown remarkable results in different machine learning tasks, learning classification, regression, and most relevant to our work, generating temporal sequences. We discuss the challenges associated with evaluating our methods, which is related to evaluation of generative models in general. We then propose evaluation metrics, which we find relevant to this problem, and we discuss how we evaluate the evaluation metrics. In this study, we use IRON-OFF dataset [1]. To the best of our knowledge, there is no work done before in generating handwriting (either in terms of methodology or the performance metrics), our in exploring styles using this dataset.

Neural Language Generation: Formulation, Methods, and Evaluation

“…Character boundaries are not always well-defined, which makes it hard to segment handwritten text into individual pieces or characters. In addition, handwriting evaluation is ambiguous and not well defined given the multitude of existent human handwriting style profiles (Mohammed et al, 2018).…”

Transfer and Extraction of the Style of Handwritten Letters using Deep Learning

“…We want metrics to capture the distance between the generated and the ground truth distributions. Following the work done in [1], we use the same two evaluation metrics in our model: ground truth letter? We keep doing this for segments of increasing length (the length of the segment here is the number of grams used in the BLEU score).…”

“…While this metric is interesting to explore, it is not directly applicable to our case, since it assumes the use of convolutional neural network. [1] also addressed the problem of evaluation of handwriting generation. They used the BLEU score [20] (a metric widely used in text translation and image captioning) and the End of Sequence (EoS) analysis.…”

“…We want metrics to capture the distance between the generated and the ground truth distributions. Following the work done in [1], we use the same two evaluation metrics in our model:…”

“…The reason we use an autoencoder is that there is no explicit way that we know about to evaluate the quality of the handwriting styles other than using them to generate handwriting, and evaluate this generation. [1] introduced benchmarks and evaluation metrics in order to assess the quality of generating handwritten letters. In comparison to the those benchmarks and metrics, we achieve higher performance, while extracting a meaningful latent space.…”