Question-based computational language assessment shows higher validity than rating scales in the categorization of emotional state narratives

Abstract

Psychological constructs are commonly quantified with closed-ended rating scales, however, recent advances in natural language processing (NLP) have been shown to allow for the quantification of open-ended language responses with unprecedented accuracy. Here, we showed that emotional events analysed by NLP show higher accuracy in categorizing emotional states than rating scales. One group of participants (N = 297) was asked to generate narratives related to four emotions – depression, anxiety, satisfaction, or harmony and to rate them on four standardized rating scales (i.e., PHQ-9, GAD-7, SWLS, and HILS). The second group of participants (N = 400 controls and N = 34 healthcare professionals), read the narratives produced by the first group, summarized them in five descriptive keywords, and rated the narratives on the same four emotional states. The descriptive words were quantified by NLP methods, and machine learning was used to categorize the responses into the corresponding emotional categories. The results showed a substantially more accurate categorization of the narratives based on descriptive words (64 %) than on rating scales (44 %), indicating that semantic measures have significantly higher predictive accuracy than the corresponding four rating scales. In conclusion, our results show that semantic responses can have a higher validity in categorizing narratives of emotions than validated Likert scales.