Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

Hommel, Björn Erik; F, Wollang; Kotova, Veronika; Zacher, Hannes; Sc, Schmukle

doi:10.31234/osf.io/qfvpe

Cited by 8 publications

(10 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Psychometrics, a sub-discipline closely related to construct definitions, has also benefited from NLP. Specifically, machine learning can facilitate the time consuming process of item generation in psychological scale development (Götz et al, 2023;Hommel et al, 2022). Generative language models are deep neural networks that produce fluent, human-like text based on a prompt, that is a short text or sentence (Radford et al, 2019).…”

Section: Methods To Reveal Constructsmentioning

confidence: 99%

A Computational Method to Reveal Psychological Constructs from Text Data

Herderich,

Freudenthaler,

Garcia

2023

Preprint

View full text Add to dashboard Cite

When starting to formalize psychological constructs, researchers traditionally rely on two distinct approaches: The quantitative approach, which defines constructs as part of a testable theory based on prior research and domain knowledge often deploying self-report questionnaires, or the qualitative approach, which gathers data mostly in the form of text and bases construct definitions on exploratory analyses. Quantitative research might lead to an incomplete understanding of the construct, while qualitative research is limited due to challenges in the systematic data processing, especially at large scale. We present a new computational method that combines the comprehensiveness of qualitative research and the scalability of quantitative analyses to define psychological constructs from semi-structured text data. Based on structured questions, participants are prompted to generate sentences reflecting instances of the construct of interest. We apply computational methods to calculate embeddings as numerical representations of the sentences, which we then run through a clustering algorithm to arrive at groupings of sentences as psychologically-relevant classes. The method includes steps for the measurement and correction of bias introduced by the data generation, and the assessment of cluster validity according to human judgment. We demonstrate the applicability of our method on an example from emotion regulation. Based on short descriptions of emotion regulation attempts collected through an open-ended situational judgment test, we use our method to derive classes of emotion regulation strategies. Our approach shows how machine learning and psychology can be combined to provide new perspectives on the conceptualization of psychological processes.

show abstract

Section: Methods To Reveal Constructsmentioning

confidence: 99%

A Computational Method to Reveal Psychological Constructs from Text Data

Herderich,

Freudenthaler,

Garcia

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In educational assessments, traditional LLMs were used, for instance, for text scoring (e.g., Flor & Hao, 2021). Another application of DL-based NLP models beyond simple text scoring is the generation of new items for assessments (Hommel, Wollang, Kotova, Zacher, & Schmukle, 2022;von Davier, 2018). In an overview paper, Yan et al (2024) list multiple use cases for applications of LLMs in education, such as grading, the support of teachers, feedback, content generation, and recommendation.…”

Section: Current Applications Of Llms In the Behavioral And Social Sc...mentioning

confidence: 99%

From Embeddings to Explainability: A Tutorial on Transformer-Based Text Analysis for Social and Behavioral Scientists

Debelak,

Koch,

Aßenmacher

et al. 2024

Preprint

View full text Add to dashboard Cite

Large language models and their use for text analysis have had a significant impact on psychology and the social and behavioral sciences in general. Key applications include the analysis of texts, such as social media posts, to infer psychological traits, as well as survey and interview analysis. In this tutorial paper, we demonstrate the use of the Python-based natural language processing software package transformers (and related modules from the huggingface universe) that allow for the automated classification of text inputs. In doing so, we rely on pre-trained transformer models which can be fine-tuned to a specific task and domain. The first proposed application of this model class is to use it as a feature extractor, allowing for the transformation of written text into real-valued numerical vectors (called "embeddings") that capture a text's semantic meaning. These vectors can, in turn, be used as input for a subsequent machine-learning model. The second presented application of transformer models is the end-to-end training (so-called "fine-tuning") of the model. This results in a direct prediction of the label within the same model that directly maps the text to the embeddings. While in the second case, results are usually better and training works more seamlessly, the model itself is often not directly interpretable. We showcase an alleviation of this issue via the application of post-hoc interpretability methods by calculating SHAP values and applying local interpretable model-agnostic explanations (LIME) in an attempt to explain the model's inner workings.

show abstract

“…Based on these trends, natural language processing (NLP) is becoming increasingly widespread in areas of psychological assessment 20,21 . The hope is that this might be a way to convey a more semantic meaning and that it could more closely represent how people view each other 22 . NLP has the potential to capture how individuals with their words relate to psychological constructs.…”

Section: Five Small Words Capture the Big Fivementioning

confidence: 99%

Personality in Just a Few Words: Assessment Using Natural Language Processing

Sikström,

Valavičiūtė,

Kajonius

2024

Preprint

View full text Add to dashboard Cite

Assessment of psychological constructs, for example, the Big Five personality traits, have till now predominantly relied on standardized rating scales. While these rating scales have several advantages, we propose that a descriptive word-based response analyzed with natural language processing (NLP) is a promising novel method for measuring personality traits. We asked participants (N = 663) to describe either their own personality or describe a person that is high in one of the big five traits, using five words. These responses were analyzed using a large language model (e.g., BERT). The main objective was to assess the validity of word response describing personality traits analyzed by NLP and compare this to the validity of a commonly used rating scale for assessment of the Big Five personality traits (i.e., IPIP-NEO-30). The average prediction accuracy for categorizing the Big Five traits, based on stimuli material with known personality traits, was up to 10% higher for descriptive word responses than for rating scales. The results also showed that semantic measures had higher inter-rater reliability and that observer convergence was higher in assessments of others than self. This suggests that descriptive word-based descriptions may capture more broad and noticeable aspects of personality than rating scales.

show abstract

Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

Cited by 8 publications

References 32 publications

A Computational Method to Reveal Psychological Constructs from Text Data

A Computational Method to Reveal Psychological Constructs from Text Data

From Embeddings to Explainability: A Tutorial on Transformer-Based Text Analysis for Social and Behavioral Scientists

Personality in Just a Few Words: Assessment Using Natural Language Processing

Contact Info

Product

Resources

About