Benchmarking online sequence-to-sequence and character-based handwriting recognition from IMU-enhanced pens

Abstract

Handwriting is one of the most frequently occurring patterns in everyday life and with it comes challenging applications such as handwriting recognition, writer identification and signature verification. In contrast to offline HWR that only uses spatial information (i.e., images), online HWR uses richer spatio-temporal information (i.e., trajectory data or inertial data). While there exist many offline HWR datasets, there are only little data available for the development of OnHWR methods on paper as it requires hardware-integrated pens. This paper presents data and benchmark models for real-time sequence-to-sequence learning and single character-based recognition. Our data are recorded by a sensor-enhanced ballpoint pen, yielding sensor data streams from triaxial accelerometers, a gyroscope, a magnetometer and a force sensor at 100 Hz. We propose a variety of datasets including equations and words for both the writer-dependent and writer-independent tasks. Our datasets allow a comparison between classical OnHWR on tablets and on paper with sensor-enhanced pens. We provide an evaluation benchmark for seq2seq and single character-based HWR using recurrent and temporal convolutional networks and transformers combined with a connectionist temporal classification (CTC) loss and cross-entropy (CE) losses. Our convolutional network combined with BiLSTMs outperforms transformer-based architectures, is on par with InceptionTime for sequence-based classification tasks and yields better results compared to 28 state-of-the-art techniques. Time-series augmentation methods improve the sequence-based task, and we show that CE variants can improve the single classification task. Our implementations together with the large benchmark of state-of-the-art techniques of novel OnHWR datasets serve as a baseline for future research in the area of OnHWR on paper.

Improving Accuracy and Explainability of Online Handwriting Recognition

“…Improving on the accuracy of classifiers on OnHW-chars To our knowledge, the best accuracy results of ML classifiers and DL classifiers for the OnHW-chars dataset are the ones reported in [27] and [26], respectively. As mentioned before, [26] improves DL classifiers in [27].…”

“…Improving on the accuracy of classifiers on OnHW-chars To our knowledge, the best accuracy results of ML classifiers and DL classifiers for the OnHW-chars dataset are the ones reported in [27] and [26], respectively. As mentioned before, [26] improves DL classifiers in [27]. Our ML and DL models yield, respectively, 11.3%-23.56% and 2.17%-4.34% improvements over the accuracy of the best ML and DL models reported in [27,26].…”

“…To our knowledge, source codes of algorithms for processing data and trained models in [27] are not publicly available. DL classifiers, trained on the same dataset OnHW-chars, have later been improved in a more recent paper [26]. In addition, [26] reports on the accuracy of classifiers trained on other new datasets, such as OnHW-equations, OnHW-symbols and OnHW-words.…”

“…DL classifiers, trained on the same dataset OnHW-chars, have later been improved in a more recent paper [26]. In addition, [26] reports on the accuracy of classifiers trained on other new datasets, such as OnHW-equations, OnHW-symbols and OnHW-words. To our knowledge, source codes of algorithms, models, and the new datasets in [26] have not been released to public yet.…”

“…In all this yields 30 datasets. When reporting on the accuracy of OnHW-chars classifiers in [27,26], they average over 5 folds. Thus, they present classifier accuracies on 6 datasets, which we denote by lowercase-WD (L-WD), lowercase-WI (L-WI), uppercase-WD (U-WD), uppercase-WI (U-WI), combined-WD (C-WD), combined-WI (C-WI).…”