Is It Biased? Empirical Analysis of Various Phenomena That Affect Survey Results

Abstract

It is often assumed that survey results reflect only the quality of the sample and the underlying measuring instruments used in the survey. However, various phenomena can affect the results, but these influences are often neglected when conducting surveys. This study aimed to test the influences of various method effects on survey results. We tested the influences of the following method effects: item wording, confirmatory bias, careless responding, and acquiescence bias. Using a split-ballot survey design with online questionnaires, we collected data from 791 participants. We tested if these method effects had an influence on mean values, item correlations, construct correlations, model fits, and construct measurement invariance. The instruments used to test these influences were from the domain of personality and gender inequality, and their items were adapted based on the method effect tested. All tested method effects, except careless responding, had a statistically significant effect on at least one component of the analysis. Item wording and confirmatory bias affected mean values, model fit, and measurement invariance. Controlling for acquiescence bias improved the fit of the model. This paper confirms that the tested method effects should be carefully considered when using surveys in research, and suggests some guidelines on how to do so.

An Examination of the Role of Inverted Dark Tetrad Items on Structural Properties and Construct Validity

“…Authors including inverted items make use of the decomposition of an observed value into true and error-related aspects (i.e., X = T + E). By reducing sources of response bias, reversed items should decrease noise (E), and thus help approximate an individual's true score, T, through the observed score, X (Mandić & Klasnić, 2021). The outlined theoretical advantage of balanced item sets addresses the frequent concerns that reversed items correlate only moderate with regular items, making positive and negative items constitute separate factors rather than a common factor.…”

Teaching Competencies and Recommendations of Croatian Teachers Abroad: A Quantitative Analysis

“…No s obzirom na to da je i sama istraživana populacija vrlo mala (ukupno 97 osoba), a ovim je istraživanjem obuhvaćeno čak 63 % te populacije, već se i deskriptivni nalazi mogu smatrati relevantnima. Nadalje, tehnika samoevaluacije rada učitelja, kao i ostalih samoiskaza putem slaganja s ponuđenim tvrdnjama, može patiti od metodoloških pristranosti, koje je u provedbi ovakva istraživanja teško kontrolirati, poput formulacije iskaza, potvrdne pristranosti, nemarnog odgovaranja te pristranosti slaganja (Mandić i Klasnić, 2021). S tim u vezi upotreba novih, dosad nevalidiranih, instrumenata u ovoj studiji i nepostojanje prethodnih istraživanja s kojima bi se naši nalazi usporedili također predstavljaju ograničenje ovog rada te upućuju na potrebu za kontinuiranim i sistematičnim praćenjem zadovoljstva i evaluacije rada učitelja HNuI.…”

A New Generation of Climate Activists?

“…Greta Thunberg ima moju punu podršku za sve što radi za zaštitu okoliša). S obzirom na to zaključili smo da su dimenzije vrlo vjerojatno rezultat postojanja čestica obrnutog smjera, a ne konceptualno različitih predmeta mjerenja, kako potvrđuju brojne metrijske studije (Mandić i Klasnić, 2021;Suárez-Álvarez et al, 2018;Marsh, 1986), te instrument u ovoj studiji tretiramo jednodimenzionalno. Konstruirana kompozitna skala ima visoku pouzdanost (α=0,842), a rezultati se interpretiraju na način da viši rezultat označava negativnije stavove o Greti Thunberg.…”