A disagreement about within-group agreement: Disentangling issues of consistency versus consensus.

Kozlowski, Steve W. J.; Hattrup, Keith

doi:10.1037/0021-9010.77.2.161

Cited by 433 publications

(393 citation statements)

References 28 publications

Supporting

Mentioning

379

Contrasting

Unclassified

Order By: Relevance

“…And yet Kozlowski and Hattrup (1992), among others, recognized that individuals working within bounded organizational contexts (e.g., teams) may encounter homogenous situational factors that lead to shared interpretations and collective response tendencies. Importantly, functional relationships at more than one level of analysis cannot be assumed equivalent (Kozlowski & Klein, 2000).…”

Section: Rationale For the Present Conceptual Schemementioning

confidence: 99%

Affective mechanisms linking dysfunctional behavior to performance in work teams: A moderated mediation study.

Cole¹,

Walter²,

Bruch³

2008

Journal of Applied Psychology

295

260

View full text Add to dashboard Cite

The present study examines the association between dysfunctional team behavior and team performance. Data included measures of teams' dysfunctional behavior and negative affective tone as well as supervisors' ratings of teams' (nonverbal) negative emotional expressivity and performance. Utilizing a field sample of 61 work teams, the authors tested the proposed relationships with robust data analytic techniques. Results were consistent with the hypothesized conceptual scheme, in that negative team affective tone mediated the relationship between dysfunctional team behavior and performance when teams' nonverbal negative expressivity was high but not when nonverbal expressivity was low. On the basis of the findings, the authors conclude that the connection between dysfunctional behavior and performance in team situations is more complex than was previously believed-thereby yielding a pattern of moderated mediation. In sum, the findings demonstrated that team members' collective emotions and emotional processing represent key mechanisms in determining how dysfunctional team behavior is associated with team performance.

show abstract

Section: Rationale For the Present Conceptual Schemementioning

confidence: 99%

Affective mechanisms linking dysfunctional behavior to performance in work teams: A moderated mediation study.

Cole¹,

Walter²,

Bruch³

2008

Journal of Applied Psychology

295

260

View full text Add to dashboard Cite

show abstract

“…With instrument reliability we refer in a broad sense to features of stability and consistency in instrument use by different raters [1]. Measuring the level of interrater agreement informs about the extent to which different raters essentially make the same assessments [2][3][4]. This fundamental aspect of instrument reliability concerns the stability of assessments across different raters.…”

Section: Introductionmentioning

confidence: 99%

“…This fundamental aspect of instrument reliability concerns the stability of assessments across different raters. For an instrument to be regarded as reliable though, it is equally important that there is a consistency among raters with respect to assessment variability [2][3][4]. To achieve high reliability, in addition to high agreement it must also be possible to detect true variation by means of the assessments [5].…”

Section: Introductionmentioning

confidence: 99%

Unfolding the phenomenon of interrater agreement: a multicomponent approach for in-depth examination was proposed

Slaug

Schilling

Helle

et al. 2012

Journal of Clinical Epidemiology

View full text Add to dashboard Cite

Objective: The overall objective was to unfold the phenomenon of inter-rater agreement: to identify potential sources of variation in agreement data and to explore how they can be statistically accounted for. The ultimate aim was to propose recommendations for in-depth examination of agreement, in order to improve the reliability of assessment instruments.Study Design and Setting: Utilizing a sample where 10 rater pairs had assessed the presence/absence of 188 environmental barriers by a systematic rating form, a raters items dataset was generated (N=1,880). In addition to common agreement indices, relative shares of agreement variation were calculated. Multilevel regression analysis was carried out, using rater and item characteristics as predictors of agreement variation.Results: Following a conceptual decomposition, the agreement variation was statistically disentangled into relative shares. The raters accounted for 6-11%, the items for 32-33% and the residual for 57-60% of the variation. Multilevel regression analysis showed barrier prevalence and raters" familiarity with using standardized instruments to have the strongest impact on agreement.Conclusion: Supported by a conceptual analysis we propose an approach of in-depth examination of agreement variation, as a strategy for increasing the level of inter-rater agreement. By identifying and limiting the most important sources of disagreement, ultimately instrument reliability can be improved.

show abstract

“…The r wg index is also not a statistical measure of similarity in the rank order of items across clinicians; instead, it measures whether or not clinicians are giving essentially the same ratings on treatment belief items. Specifically, the r wg index represents the proportion of observed variance compared to the proportion of variance one may expect with random responding on the item (Kozlowski & Hattrup, 1992). Typically, an r wg of .70 or greater is treated as evidence that there is sufficient agreement among raters to justify aggregating responses to a group level, i.e., that a particular perception can be treated as shared at the group level of analysis within a specific aggregation of individuals (Klein & Kozlowski, 2000).…”

Section: Analysesmentioning

confidence: 99%

“…These less conservative analyses are useful for determining whether a "shared vision" exists, and if so, at what level of grouping (i.e., organization, recovery status, methadone clinic, or research affiliation) it exists. To make these assessments, we relied on the r wg index, which is a measure of agreement or consensus within a group (Kozlowski & Hattrup, 1992;James, Demaree, & Wolf, 1993). In the current study, r wg represents the degree of similarity among clinics, in recovery/not in recovery clinicians' ratings of treatment beliefs, methadone/non-methadone clinic, and research/non-research affiliated clinic, or the extent to which these groups of clinicians give the same or very similar ratings to the treatment belief items.…”

Section: Analysesmentioning

confidence: 99%

Level of analysis issues in assessing treatment beliefs in substance abuse clinics

Mitchelson

Dickson

Arfken

et al. 2007

Addictive Behaviors

View full text Add to dashboard Cite

The current study applies the growing literature in the organizational sciences regarding levels of analysis issues to the analysis of substance abuse treatment beliefs. Research on clinicians' beliefs in substance abuse treatment is often based on the assumption that the beliefs are sufficiently shared by clinicians within a clinic and sufficiently vary across clinics that they can be treated as a group-level phenomenon. Further, efforts to introduce new innovations are often focused at the group or clinic level without testing this assumption, which can lead to failure to adopt or to successfully implement the innovation. We tested the assumption of sharedness by examining if there was sufficient agreement about treatment beliefs within clinics, within groups of clinics or within groups of clinicians to justify treating these aggregations as meaningful groups. Using three statistical approaches to examining level of analysis (Within and Between Analysis (WABA I), Intraclass Correlation Coefficients (ICC(1)), and r wg ), we found that variability in treatment beliefs largely occurred at the individual rather than at the tested aggregate levels of analysis. These findings serve as an example of the importance of testing the assumption of shared perceptions in future research.

show abstract

A disagreement about within-group agreement: Disentangling issues of consistency versus consensus.

Cited by 433 publications

References 28 publications

Affective mechanisms linking dysfunctional behavior to performance in work teams: A moderated mediation study.

Affective mechanisms linking dysfunctional behavior to performance in work teams: A moderated mediation study.

Unfolding the phenomenon of interrater agreement: a multicomponent approach for in-depth examination was proposed

Level of analysis issues in assessing treatment beliefs in substance abuse clinics

Contact Info

Product

Resources

About