The failing measurement of attitudes: How semantic determinants of individual survey responses come to replace measures of attitude strength

Abstract

The traditional understanding of data from Likert scales is that the quantifications involved result from measures of attitude strength. Applying a recently proposed semantic theory of survey response, we claim that survey responses tap two different sources: a mixture of attitudes plus the semantic structure of the survey. Exploring the degree to which individual responses are influenced by semantics, we hypothesized that in many cases, information about attitude strength is actually filtered out as noise in the commonly used correlation matrix. We developed a procedure to separate the semantic influence from attitude strength in individual response patterns, and compared these results to, respectively, the observed sample correlation matrices and the semantic similarity structures arising from text analysis algorithms. This was done with four datasets, comprising a total of 7,787 subjects and 27,461,502 observed item pair responses. As we argued, attitude strength seemed to account for much information about the individual respondents. However, this information did not seem to carry over into the observed sample correlation matrices, which instead converged around the semantic structures offered by the survey items. This is potentially disturbing for the traditional understanding of what survey data represent. We argue that this approach contributes to a better understanding of the cognitive processes involved in survey responses. In turn, this could help us make better use of the data that such methods provide.

Measuring semantic components in training and motivation: A methodological introduction to the semantic theory of survey response

“…Individual respondents may of course not always be as systematic or coherent in their responses, but the statistics of a sample will quite quickly converge around the semantically expected patterns, as happened in the present case. A recent study on STSR that separated attitude strength from semantic components in survey data found that correlation matrices are sometimes completely void of information about attitude strength, leaving only traces of semantics in the models (Arnulf, Larsen, & Martinsen, ; Arnulf, Larsen, Martinsen, et al, ).…”

“…Second, it may be that semantic relationships are somehow different, twisted, or disregarded by certain subsamples. Personality, demographics, professional training, and interest in the subject matter are examples of factors that seem to affect the degree to which respondents comply with what is semantically given (Arnulf, Larsen, Martinsen, et al, ). LSA has already been used to identify schizophrenic from nonschizophrenic subjects in psychiatric diagnoses based on the patients' level of semantic predictability (Elvevag et al, ).…”

“…The present dataset is a previously published work on the relationship between perceptions of training, IM, and work effort (WE) and work quality (WQ) (Dysvik et al, ). We use this dataset for two main reasons: Primarily, we know that some of the constructs in the survey are semantically predictable, because all the constructs except perception of training have been used in previous publications on STSR (Arnulf et al, ; Arnulf, Larsen, Martinsen, et al, ). By expanding this set of constructs with perceptions of training, we can bridge the research on semantics across disciplines such as leadership, OB, and HRD.…”

“…We want to point out that the findings on semantics have so far raised a series of questions about the nature of survey data and what the purported measures really “measure” (Arnulf, Larsen, Martinsen, et al, ; Mari, Maul, Irribarra, & Wilson, ; Maul, ; Slaney, ). Working with semantic data in this domain is, therefore, fraught with unsettled questions and potential conflicting views about what these numbers represent.…”

“…Semantic algorithms are also able to predict factor structures in surveys on technology acceptance as well as identifying and separating overlapping constructs in social science in general (Larsen & Bong, 2016). These findings are troublesome to the traditional understanding of the survey data as stemming from attitude strength (Likert, 1932) because the aggregated correlation matrices can be shown to have filtered out the attitude strength of individual respondents, leaving only the semantically predictable information in the models (Arnulf, Larsen, Martinsen, & Egeland, 2018). In the case of surveys with high semantic coherence, the semantic information allows the prediction of individual responses with high probability .…”