Scenario-Based Methods for Evaluating Collaborative Systems

Abstract

Evaluating collaborative systems remains a significant challenge. Most evaluation methods approach the problem from one of two extremes: focused evaluation of specific system features, or broad ethnographic investigations of system use in context. In this paper, we develop and demonstrate a middle ground for evaluation: explicit reflections on scenarios of system use coupled with analysis of the consequences of these use scenarios, represented as claims. Extending prior work in scenario-based design and claims analysis, we develop a framework for a multiperspective, multi-level evaluation of collaborative systems called SWIMs: scenario walkthrough and inspection methods. This approach is centered on the capture, aggregation, and analysis of users' reflections on system support for specific scenarios. We argue that this approach not only provides the contextual sensitivity and use centricity of ethnographic techniques, but also sufficient structure for method replication, which is common to more feature-based evaluation techniques. We demonstrate with an extensive field study how SWIMs can be used for summative assessment of system performance and organizational contributions as well as formative assessment to guide system and feature re-design. Results from the field study provide preliminary indications of the method's effectiveness and suggest directions for future research.

“Social credit effect” in a sharing economy: A theory of mind and prisoner's dilemma game theory perspective on the two‐way review and rating system

“…A scenario‐based methodology was used to integrate and illustrate the set of impacts of the two‐way evaluation system that could act as examples of context to evaluate Uber passengers' thoughts dependent on the star rating received. In a similar vein with Haynes, Purao, and Skattebo () and Amatulli, Guido, and Nataraajan (), we argue that this approach provides the contextual sensitivity of the research method and also provides a sufficient structure for method replication, which is common in more feature‐based, quantitative evaluation techniques. This form of data collection was also chosen as it puts respondents in their possible realities (Stewart, Shamdasani, & Rook, ).…”

Context Effects on Older Consumers’ Cognitive Age: The Role of Hedonic versus Utilitarian Goals

“…This technique has a long tradition in consumer research on behavioral intentions and expectancy‐value models (e.g., Ajzen, ; Fishbein & Ajzen, ). In line with Haynes, Purao, and Skattebo (), it was argued that such an approach provides not only the contextual sensitivity of qualitative techniques, but also a sufficient structure for method replication, which is common to more feature‐based, quantitative evaluation techniques. Representative scenarios, generated from responses that were most frequently recorded in a series of focus groups, were selected in this study, identifying features of consumption situations that might influence the age one might feel.…”

Does revealing contextual knowledge of the patient’s intention help nurses’ handling of nurse calls?

“…The scenarios are 'stories about people and their activities' [33] that incorporate the use of technology, with a purpose of (1) presenting and situating solutions, (2) illustrating alternative solutions and (3) identifying potential problems [32]. An important aspect is that the scenarios should be based on a good understanding of the context for which the system is developed [32,34,35].…”