“Transforming” Personality Scale Development: Illustrating the Potential of State-of-the-Art Natural Language Processing

Abstract

Natural language processing (NLP) techniques are becoming increasingly popular in industrial and organizational psychology. One promising area for NLP-based applications is scale development; yet, while many possibilities exist, so far these applications have been restricted—mainly focusing on automated item generation. The current research expands this potential by illustrating an NLP-based approach to content analysis, which manually categorizes scale items by their measured constructs. In NLP, content analysis is performed as a text classification task whereby a model is trained to automatically assign scale items to the construct that they measure. Here, we present an approach to text classification—using state-of-the-art transformer models—that builds upon past approaches. We begin by introducing transformer models and their advantages over alternative methods. Next, we illustrate how to train a transformer to content analyze Big Five personality items. Then, we compare the models trained to human raters, finding that transformer models outperform human raters and several alternative models. Finally, we present practical considerations, limitations, and future research directions.

Measuring the menu, not the food: “psychometric” data may instead measure “lingometrics” (and miss its greatest potential)

“…At the moment of writing, studies demonstrating semantically predictable research findings and picking up at an increasing pace covering state-of-the-art research instruments in leadership and motivation ( Arnulf et al, 2014 ), engagement, job-satisfaction and well-being ( Nimon et al, 2016 ), the technology acceptance model ( Gefen and Larsen, 2017 ), job analysis ( Kobayashi et al, 2018 ), personality scale construction ( Abdurahman et al, 2023 ; Fyffe et al, 2023 ), entrepreneurship ( Freiberg and Matz, 2023 ) personality and mental health ( Kjell K. et al, 2021 ; Kjell O. et al, 2021 ) or even near-death-experiences ( Lange et al, 2015 ). Overlapping meanings between a vast group of constructs have been demonstrated ( Larsen and Bong, 2016 ) and new scales can be checked for overlaps ( Rosenbusch et al, 2020 ; Nimon, 2021 ).…”

“…It is important to understand that NLP technologies do not only map and compute wordings of questionnaires, but their calculations also pervade definitions of variables and constructs ( Fyffe et al, 2023 ; Larsen et al, 2023 ). Since these calculations span the scientific process from empirically collected respondent data to the theoretical argumentation of the researchers, we need to reconsider the distinction between empirical and semantic features of data.…”

Language models accurately infer correlations between psychological items and scales from text alone

“…Pre-transformer era attempts to use semantic features of items to predict associations between measurement scales using latent semantic analysis have demonstrated moderate utility (Arnulf et al, 2014;Larsen & Bong, 2016;Rosenbusch et al, 2020). As the ability of computerised language models to capture meaning has grown, researchers have sought to directly quantify relationships between adjectives from textual data (Cutler & Condon, 2022), to assign items to constructs (Fyffe et al, 2024;Guenole et al, 2024), to directly predict item responses (Abdurahman et al, 2024;Argyle et al, 2023) and quantify openended answers to questions (Kjell et al, 2019(Kjell et al, , 2024. used large language models (LLMs) to map survey items to vector space and predict empirical item correlations.…”

Call for Papers for a Feature Topic: Having A Way with Words: Innovations and Improvements in Text Analysis Methods

“…First, we aim to encourage the introduction and dissemination of new and innovative techniques that have the potential to inform organizational research, enabling researchers to provide stronger tests of existing research questions and opening the door to research questions that are difficult to test with current tools. For example, previous work has introduced innovations in text analysis such as topic modeling (e.g., Schmiedel et al, 2019), transformer-based text classification (e.g., Fyffe et al, 2023), and CATA dictionary development (e.g., Short et al, 2010) to organizational research, demonstrating how each improves upon current practices in text analysis.…”