Predictive composition of pictogram messages for users with autism

Abstract

Communication is a basic need for every person. However, there are many people who present disabilities that prevent communication through natural language. Augmentative and Alternative Communication (AAC) systems, including those based on pictograms, attempt to facilitate the communication for people with this kind of difficulties. In this paper we present PictoEditor, an Augmentative and Alternative Communication application for the composition of pictogram messages for users with autism that incorporates prediction functionalities. Although such functionalities have been widely studied in text-based Augmentative and Alternative Communication tools, they have not been applied to pictogram based ones. The results show that prediction based on frequency of use of specific pictograms improves the immediate availability of the desired pictograms, but the improvement with pre-The work presented in this paper has been funded by the projects IDiLyCo: Digital Inclusion, Language and Communication, Grant. No.

Pictogram Prediction in Alternative Communication Boards: a Mapping Study

“…For this reason, we opted to maintain their label classification separated instead of grouping them as knowledge bases. Two studies using statistical language models based on n-grams [Hervás et al 2020, Garcia et al 2016. These studies trained bi-gram language models by using pre-defined text corpora.…”

“…The student is a child with complex communication needs. In [Hervás et al 2020], a teacher working with autistic children with complex communication needs participated in the experiments, which involved reproducing the children's conversations in the class over five weeks in the AAC tool. This way, most of the studies used an automatic evaluation and assessed their proposal quality without the participation of actual AAC users.…”

“…[Saturno et al 2015] assessed the number of pictograms used by the experiment participant to construct the proposed sentences and the time spent. [Hervás et al 2020], which also performed a quasi-experiment, used the same metrics and a top-n accuracy, which indicates whether the pictogram the participant used was on top-n predicted by the model. Top-n accuracy was also used by [Dudy and Bedrick 2018] and [Pereira et al 2022], the two approaches based on deep learning.…”

“…As discussed in Section 1, in high-tech AAC systems, each pictogram has an associated label or caption, which can be a word or a multi-word expression. Some of the analyzed studies consider this label enough for making pictogram prediction [García et al 2015, Saturno et al 2015, Garcia et al 2016, Hervás et al 2020. This way, they perform a word prediction and do not take care of polysemic words.…”

“…For example, the English word "bat" can have many meanings (e.g., "nocturnal mouselike mammal" or "a club used for hitting a ball") and, similarly, many related pictograms in a given vocabulary. In addition to the label, [Hervás et al 2020] opted to use its POS tag (e.g., verb, noun, or adjective) as a prediction unit. They trained a bi-gram language model using the sequence of POS tags as training data.…”