J. N. Cleveland and K. R. Murphy (1992) suggested that phenomena such as rater errors and interrater disagreements could be understood in terms of differences in the goals pursued by various raters. We measured 19 rating goals of students at the beginning of a semester, grouped them into scales, and correlated these with teacher evaluations collected at the end of the semester. We found significant multiple correlations, both within classes and in an analysis of the pooled sample (adjusting for instructor mean differences, incremental R2 =.08). Measures of rating goals obtained after raters had observed a significant proportion of ratee performance accounted for variance (incremental R2 =.07) not accounted for by measures of goals obtained at the beginning of the semester.

We conducted a laboratory study examining the effect of a family conflict with work on performance appraisal ratings given to men and women. Overall, the experience of a family conflict was associated with lower performance ratings, and ratee sex moderated this relationship. Men who experienced a family conflict received lower overall performance ratings and lower reward recommendations than men who did not, whereas ratings of women were unaffected by the experience of a family conflict. The sex bias was not evident when performance was evaluated on the more specific dimension of planning. Neither rater gender nor work-family role attitudes moderated the sex bias. Implications for future research and practice are discussed.

We report on the use of scenario-based methods for evaluating collaborative systems. We describe the method, the case study where it was applied, and provide results of its efficacy in the field. The results suggest that scenariobased evaluation is effective in helping to focus evaluation efforts and in identifying the range of technical, human, organizational, and other contextual factors that impact system success. The method also helps identify specific actions such as prescriptions for design to enhance system effectiveness. However, we found the method somewhat less useful for identifying the measurable benefits gained from a CSCW implementation, which was one of our primary goals. We discuss challenges faced applying the technique, suggest recommendations for future research, and point to implications for practice.

Evaluating collaborative systems remains a significant challenge. Most evaluation methods approach the problem from one of two extremes: focused evaluation of specific system features, or broad ethnographic investigations of system use in context. In this paper, we develop and demonstrate a middle ground for evaluation: explicit reflections on scenarios of system use coupled with analysis of the consequences of these use scenarios, represented as claims. Extending prior work in scenario-based design and claims analysis, we develop a framework for a multiperspective, multi-level evaluation of collaborative systems called SWIMs: scenario walkthrough and inspection methods. This approach is centered on the capture, aggregation, and analysis of users' reflections on system support for specific scenarios. We argue that this approach not only provides the contextual sensitivity and use centricity of ethnographic techniques, but also sufficient structure for method replication, which is common to more feature-based evaluation techniques. We demonstrate with an extensive field study how SWIMs can be used for summative assessment of system performance and organizational contributions as well as formative assessment to guide system and feature re-design. Results from the field study provide preliminary indications of the method's effectiveness and suggest directions for future research.

No abstract