Manipulating Common Method Variance via Experimental Conditions

Abstract

Research data collected from single respondents may raise concerns regarding common method variance (CMV), which is believed to threaten the validity of findings. The primary concern is that CMV can inflate substantive relationships, such that they appear statistically significant when they are not. Thus, understanding the nature of CMV is critical, especially when one considers the popularity—and sometimes necessity—of using self-report data. Research examining CMV has found conflicting evidence about the impact of CMV. Researchers who believe CMV influences findings have proposed solutions to combat any real or perceived potential bias, including changing survey instructions and using marker variables, but few studies have examined the efficacy of these approaches. The purpose of this study is to examine the impact of these techniques and the nature of CMV using an experimental design. To conduct the experiment, multiple versions of a survey, which vary in their use of the remedial approaches, are utilized to collect data, which resulted in 1,069 usable responses. The experimental design was based on the faking literature and included instructions intended to induce or reduce the levels of CMV. Further, two different marker variables are used to determine the degree to which they create a psychological separation in substantive variables. Correlation analysis and measurement invariance are used to analyze the data. This study posits that, if CMV is a substantial concern for self-report data and these approaches are effective, then findings will differ in surveys that incorporate such approaches from surveys that do not. Results indicate few differences in experimental conditions, meaning that regardless of instructions or marker variable, substantive item correlations remained statistically similar. The results indicate this is likely due to the minimal impact of CMV, given that the proposed methods of correction did not significantly influence research findings. These findings have implications for researchers in that they do not support that CMV, or at least its proposed remedies, significantly alter findings. However, support for the null conclusions, in spite of appropriate statistical power, warrant future research examining the nature and impact of CMV.

The relation between leadership styles in higher education institutions and academic staff’s job satisfaction: A meta-analysis study

“…Common method bias is also the case when a researcher creates estimates of validity and reliability that may lead a researcher to believe that a scale does not accurately reflect an implicit measure but does so accurately. Such an error may cause common method bias in future meta-analysis studies ( Wall, 2014 ). As a result, the studies included in the meta-analysis were examined and it was seen that the data collection tools used were suitable for the purpose of the studies, and the validity and reliability information was presented.…”

Examining the relationships among managerial coaching, perceived organizational support, and job engagement in the US higher education context

“…The two-phase survey comprised previously empirically validated instruments. The study also included a latent marker variable (Williams et al , 2010) that had been previously used with the survey of SPOS and JES (Jones, 2015; Wall, 2014).…”

“…CFA marker. The attitude toward the color blue measure (Miller and Chiodo, 2008) is an eight-item scale developed specifically to be used as a latent marker variable (Williams et al , 2010) and has previously been used alongside the SPOS and JES (Jones, 2015; Wall, 2014). Respondents rate each question on a seven-point Likert-type scale ranging from “strongly disagree” to “strongly agree.” Sample items include “I prefer blue to other colors” and “I think blue cars are ugly.” Cronbach’s alpha was 0.84 in this study.…”

“…Sample items include "I exert and lot of energy on my job," "I am CFA marker. The attitude toward the color blue measure (Miller and Chiodo, 2008) is an eight-item scale developed specifically to be used as a latent marker variable (Williams et al, 2010) and has previously been used alongside the SPOS and JES (Jones, 2015;Wall, 2014). Respondents rate each question on a seven-point Likert-type scale ranging from "strongly disagree" to "strongly agree."…”

Attitude Toward the Color Blue: An Ideal Marker Variable

“…items and guaranteeing respondent anonymity (Podsakoff et al, 2003;Podsakoff & Organ, 1986), these potential remedies have been rarely studied empirically, and the small amount of research that addresses them has shown little in the way of efficacy (Castille et al, 2017;Wall et al, 2015). In contrast, post hoc efforts at detecting CMV have been studied more extensively.…”