Applying Attention-Based Models for Detecting Cognitive Processes and Mental Health Conditions

Abstract

According to the psychological literature, implicit motives allow for the characterization of behavior, subsequent success, and long-term development. Contrary to personality traits, implicit motives are often deemed to be rather stable personality characteristics. Normally, implicit motives are obtained by Operant Motives, unconscious intrinsic desires measured by the Operant Motive Test (OMT). The OMT test requires participants to write freely descriptions associated with a set of provided images and questions. In this work, we explore different recent machine learning techniques and various text representation techniques for facing the problem of the OMT classification task. We focused on advanced language representations (e.g, BERT, XLM, and DistilBERT) and deep Supervised Autoencoders for solving the OMT task. We performed an exhaustive analysis and compared their performance against fully connected neural networks and traditional support vector classifiers. Our comparative study highlights the importance of BERT which outperforms the traditional machine learning techniques by a relative improvement of 7.9%. In addition, we performed an analysis of how the BERT attention mechanism is being modified. Our findings indicate that the writing style features acquire higher importance at the moment of accurately identifying the different OMT categories. This is the first time that a study to determine the performance of different transformer-based architectures in the OMT task is performed. Similarly, our work propose, for the first time, using deep supervised autoencoders in the OMT classification task. Our experiments demonstrate that transformer-based methods exhibit the best empirical results, obtaining a relative improvement of 7.9% over the competitive baseline suggested as part of the GermEval 2020 challenge. Additionally, we show that features associated with the writing style are more important than content-based words. Some of these findings show strong connections to previously reported behavioral research on the implicit psychometrics theory.

Using a self-attention architecture to automate valence categorization of French teenagers’ free descriptions of their family relationships. A proof of concept

“…A work based on self-attentional architectures described the use of such NLP models to classify 209000 texts from the GermEval 2020 task [12]. Pieces of free texts describing a person situation, feelings and actions from simple drawings of the Operant Motive Test [13] were classified by trained psychologists into five possible motives and rated into six possible levels.…”

“…Similarly, a close examination of BERT's attentional weights in the aforementioned GermEval classification task shows that the transformer pays more attention to form features (i.e. use of personal pronouns, stop words, negation, punctuation marks, unknown words, and some conjugation styles) than to content words [12]. While interpretability of attentional weights proves difficult, other authors have conducted a layer by layer examination of the structure of transformers by probing the corresponding hidden outputs.…”

“…Camem-BERT was used without fine-tuning. The teenagers' lexicon was not modified beyond some typos corrections, in order to stay as close as possible to free text writing without complex preprocessing, and to evaluate attentional models robustness when taking into account this population's specific style (this approach was also used in [12]). It is interesting to note that transformers' architectures per se do not take into account word orders.…”