IRT-Based Assessments of Rater Effects in Multiple-Source Feedback Instruments

Abstract

This study compares three item response theory-based models of assessing measurement equivalence in 360°feedback: the traditional differential item functioning (DIF) methodology, Muraki's rater's effect model, and Patz, Junker, and Johnson's hierarchical rater model. Using data from 491 managers collected on the Benchmarks instrument, the authors found that the traditional DIF methodology provides the most information about the rater's conception of the ratee's ability, whereas the other two models provide explicit estimates of rater leniency/severity. The authors also found that rater source effects of leniency and severity, even though statistically significant, did not substantially affect the observed score at the item and scale levels. The different results and conclusions produced by each model are discussed.

Self–other rating agreement in leadership: A review

“…In five of their samples, Lance and Bennett found that ME/I did not exist across rating groups. Similarly, several studies using IRTbased methods (e.g., Barr & Raju, 2003) found evidence of a lack of ME/I for multiple items comprising multisource assessments. In short, these studies provided somewhat mixed results, although they implied that ME/I can often be found in multisource ratings.…”

“…4.5.3. Measurement equivalence/invariance (ME/I) ME/I can be presumed to exist in multisource ratings when measures display very similar psychometric properties across rating groups (Barr & Raju, 2003). When this occurs, one can conclude that differences in ratings between rating sources are not due to, or created by, the multisource instrument itself.…”

“…This in turn gives researchers more confidence that differences in ratings can be meaningfully and accurately interpreted as true differences in how rating groups perceive target leaders (Woehr, Sheehan, & Bennett, 2005). Although ME/I is sometimes assessed via item response theory (IRT) techniques (Barr & Raju;Facteau & Craig, 2001;Meade & Lautenschlager, 2004) and logistic regression (Penny, 2003), in organizational research, it is most commonly assessed using multiple-group confirmatory factor analysis (CFA; Vandenberg & Lance, 2000;.…”

The Effects of Rater Severity and Rater Distribution on Examinees' Ability Estimation for Constructed‐response Items

“…The estimation method used is the estimation and maximization (EM) algorithm for solving marginal likelihood equations. Muraki's rater effect model is expressed as follows (Barr & Raju, 2001):…”

The development of a multirater instrument for assessing employee problem‐solving skill

“…However, to ensure the reliability of a problem-solving assessment, self-assessments should be triangulated with the assessments of others whose opinions are valued (Atkins & Wood, 2002;Miles & Huberman, 1994). Obtaining information from several sources provides multiple perspectives about an employee's skill in solving work problems and thereby strengthens the credibility of the information gathered and an employee's motivation to act on the feedback and change his or her behaviors (Barr & Raju, 2003).…”