Human Language Modeling

Soni, Nikita; Matero, Matthew; Balasubramanian, Niranjan; Schwartz, H.

doi:10.18653/v1/2022.findings-acl.52

Cited by 4 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is possible to analyze at the sentence level but still include the context of the surrounding sentences and the specific participant. We tested a Human-Aware Recurrent Transformer (HaRT; Soni et al, 2022) LLM, a model with a modified GPT-based architecture that enables processing language in the context of each participant. This model learns a recurrently updating latent participant representation (user state) derived from the language of each participant.…”

Section: Sentence Embeddingsmentioning

confidence: 99%

Automatic implicit motives codings are as accurate as humans’, cheaper, and 99% faster

Nilsson,

Runge,

Kjell

et al. 2024

Preprint

View full text Add to dashboard Cite

Implicit motives, non-conscious needs that influence individuals’ behavior and shape their emotions, have been part of personality research for nearly a century but are divergent from personality traits. The implicit motive assessment is very resource-intensive, involving expert-coding individuals’ written stories about ambiguous pictures, which has hampered implicit motive research. Using Large Language Models and machine learning techniques, we aimed to create high-quality implicit motive models that are easy for researchers to use. We trained models to code the need for power, achievement, and affiliation (85,028 sentences). The cross-validated person-level predictions converged strongly with the human coders in the training set (ICC = .83, .86, and .88 for achievement, power, and affiliation, respectively) and generalized well to the test set (ICC = .85, .87, and .89 for achievement, power, and affiliation, respectively). We demonstrated causal validity by replicating two classical experimental studies that aroused implicit motives. Finally, we let three coders re-code sentences where our implicit motives models and the original coders strongly disagreed. We found that the new coders agreed with our models in 83% of the cases (p < .001, φ = .67). Models with such high quality can be used in complement, or instead of, human coders. We provide a free, user-friendly framework in the established R-package text and a tutorial for researchers to apply the models to their data, which reduces the coding time by over 99% and requires no cognitive effort. We hope this coding automation will facilitate a historical implicit motive research renaissance.

show abstract

Section: Sentence Embeddingsmentioning

confidence: 99%

Automatic implicit motives codings are as accurate as humans’, cheaper, and 99% faster

Nilsson,

Runge,

Kjell

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The typical focus in NLP is to model language itself using huge amounts of data and to employ these language models to solve language tasks (e.g., GLUE-tasks). However, human-level AI models the individual behind the language (54,55). Modeling a person behind a text may include assessing their depression or suicide risk (54).…”

Section: Leveraging Big Data Information For Small Samplesmentioning

confidence: 99%

Beyond Rating Scales: With Targeted Evaluation, Language Models are Poised for Psychological Assessment

Kjell¹,

Kjell²,

Schwartz³

2023

Preprint

View full text Add to dashboard Cite

Artificial intelligence-based (AI-based) language analysis has been undergoing a purported “paradigm shift” initiated by new machine learning models, large language models. These models, such as GPT3 or BERT, have led to unprecedented accuracies over most computerized language processing tasks such as web search, automatic machine translation, and question answering, while their chat-based forms like ChatGPT have captured the interest of over a million users. The success of the large language model is mostly attributed to its capability to numerically represent words in their context, long a weakness of popular language assessment techniques that use “bag-of-words” or word count approaches. While many potential applications, from therapy to health information education, are beginning to be studied on the heels of chatGPT’s success, here we suggest these models are already ready to improve how we do psychological assessment. We present the case for AI's paradigm shift to large language models to be the missing piece for a parallel transformation in mental health assessment from the strong reliance on rating scale responses.

show abstract

“…Recent works (Lynn et al, 2020;Matero et al, 2021b;Soni et al, 2022) have highlighted the importance of incorporating author context into the message representations through the use of history and multi-level modeling. We use the Human aware Recurrent Transformer model (Soni et al, 2022) which is built on GPT2 (Radford et al, 2019), to produce message representations that encode the latent representation of the author as well. We adapted HaRT in two ways.…”

Section: Task Amentioning

confidence: 99%

“…Here, we explore two types of modeling techniques that can capture changes over time: Humanaware Recurrent Transformers (Soni et al, 2022) and difference embeddings. These techniques were used as part of the WWBP-SQT-lite 1 system for the CLPsych 2022 shared tasks (Tsakalidis et al, 2022a): (Task A) modeling user state changes over time (Tsakalidis et al, 2022b), and (Task B) the suicide risk associated with the user (Shing et al, 2018), our contributions are as follows: (a) evaluation of Human-aware Recurrent Transformers (HaRT) and difference embeddings for Task A (b) exploring SoTA methods for predicting state escalations and switches, and (c) exploring theoretically related linguistic assessments.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

WWBP-SQT-lite: Multi-level Models and Difference Embeddings for Moments of Change Identification in Mental Health Forums

Ganesan¹,

Varadarajan²,

Mittal³

et al. 2022

Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology

Self Cite

View full text Add to dashboard Cite

Psychological states unfold dynamically; to understand and measure mental health at scale we need to detect and measure these changes from sequences of online posts. We evaluate two approaches to capturing psychological changes in text: the first relies on computing the difference between the embedding of a message with the one that precedes it, the second relies on a "human-aware" multi-level recurrent transformer (HaRT). The mood changes of timeline posts of users were annotated into three classes, 'ordinary,' 'switching' (positive to negative or vice versa) and 'escalations' (increasing in intensity). For classifying these mood changes, the difference-between-embeddings technique -applied to RoBERTa embeddings -showed the highest overall F1 score (0.61) across the three different classes on the test set. The technique particularly outperformed the HaRT transformer (and other baselines) in the detection of switches (F1 = .33) and escalations (F1 = .61). Consistent with the literature, the language use patterns associated with mental-health related constructs in prior work (including depression, stress, anger and anxiety) predicted both mood switches and escalations.

show abstract

Human Language Modeling

Cited by 4 publications

References 32 publications

Automatic implicit motives codings are as accurate as humans’, cheaper, and 99% faster

Automatic implicit motives codings are as accurate as humans’, cheaper, and 99% faster

Beyond Rating Scales: With Targeted Evaluation, Language Models are Poised for Psychological Assessment

WWBP-SQT-lite: Multi-level Models and Difference Embeddings for Moments of Change Identification in Mental Health Forums

Contact Info

Product

Resources

About