How Incorporating Feedback Mechanisms in a DSS Affects DSS Evaluations

Abstract

M odel-based decision support systems (DSS) improve performance in many contexts that are data-rich, uncertain, and require repetitive decisions. But such DSS are often not designed to help users understand and internalize the underlying factors driving DSS recommendations. Users then feel uncertain about DSS recommendations, leading them to possibly avoid using the system. We argue that a DSS must be designed to induce an alignment of a decision maker's mental model with the decision model embedded in the DSS. Such an alignment requires effort from the decision maker and guidance from the DSS. We experimentally evaluate two DSS design characteristics that facilitate such alignment: (i) feedback on the upside potential for performance improvement and (ii) feedback on corrective actions to improve decisions. We show that, in tandem, these two types of DSS feedback induce decision makers to align their mental models with the decision model, a process we call deep learning, whereas individually these two types of feedback have little effect on deep learning. We also show that deep learning, in turn, improves user evaluations of the DSS. We discuss how our findings could lead to DSS design improvements and better returns on DSS investments.

Explaining Data-Driven Document Classifications

“…In order to examine more carefully why explanations are needed and their impact on decision model understanding, long-term learning, and improved decision making, we turn to the recent work by Kayande et al (2009). This work focuses on the same context as we do in our case study, specifically where data are voluminous, the link between decisions and outcomes is probabilistic, and the decisions are repetitive.…”

“…Prior work has suggested that when users do not understand the workings of the DS model, they will be skeptical and reluctant to use the model, even if the model is known to improve decision performance, see, e.g., Umanath and Vessey (1994), Limayem and De Sanctis (2000), Lilien et al (2004), Arnold et al (2006), Kayande et al (2009). Further, decision makers need impetus to change their decision strategies (Todd and Benbasat 1999), as well as guidance in making decisions (Silver 1991).…”

“…As pointed out by Gregor and Benbasat (1999), a user will want an explanation when she perceives an anomaly. The resultant explanation may help the user to learn about how the world works (Kayande et al 2009), and thereby improve acceptance. However, it alternatively may lead to the identification of a flaw in the model, and lead to a development effort focused on improving the model.…”

Beyond Brainstorming: Exploring Convergence in Teams

“…Structuring interventions are bundles of instructions, based on human or information technology (IT), for team members to engage in productive actions and to restrict unproductive actions during the execution of a collaborative activity [18]. Such interventions have also been referred to as facilitation [4,93], feedback [49,50,52], or coaching [91]. Structuring interventions can be directed at the process [89], the task [30], or how to use supporting technology [41].…”

“…First, most past studies have focused on structuring interventions that were targeted at the collaboration process as a whole, not solely on the convergence activity. For example, past research has examined convergence (1) in connection with generation and evaluation [38,89,93,94], (2) in larger collaboration settings considering additional activities such as problem analysis, organization, and building commitment [4,32,54,66], or (3) did not specify distinct collaboration activities [50,52,59,61,80,91]. When applied to multiple activities or to the collaboration process as a whole, the structuring intervention is not conceptually connected to a specific activity outcome.…”