Professional Judgments and “Naturalistic Decision Making”

Ross, Karol G.; Shafer, Jennifer L.; Klein, Gary

doi:10.1017/cbo9780511816796.023

Cited by 77 publications

(28 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ross et al 2006). When assessing student performance in patient encounters, experienced GP-raters paid (significantly) more attention to task-specific cues.…”

Section: Discussionmentioning

confidence: 99%

Workplace-based assessment: raters’ performance theories and constructs

Govaerts

Wiel

Schuwirth

et al. 2012

Adv in Health Sci Educ

160

155

View full text Add to dashboard Cite

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters’ implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters’ information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

show abstract

“…Ross et al 2006). When assessing student performance in patient encounters, experienced GP-raters paid (significantly) more attention to task-specific cues.…”

Section: Discussionmentioning

confidence: 99%

Workplace-based assessment: raters’ performance theories and constructs

Govaerts

Wiel

Schuwirth

et al. 2012

Adv in Health Sci Educ

160

155

View full text Add to dashboard Cite

show abstract

“…Our results significantly extend those findings by providing rich details on the kinds of procedural and substantive complexity that can inform such models when the subjects of study are seasoned professionals who are given extended and structured opportunities to reveal their thinking. Such richness and detail is entirely consistent with the mental models of experts from other domains of real‐world practice that involve ill‐structured problems, shifting, ill‐defined, or competing goals, action‐feedback loops, multiple players, and organizational goals and norms (Endsley, 2006; Ross et al., 2006). The working mental model revolves around the selection of holistic scripts, not individual tactics.…”

Section: Discussionmentioning

confidence: 99%

“…Naturalistic decision‐making research has demonstrated the value of using the rich mental models of domain experts in the training of novices (Feltovich et al., 2006; Ross et al., 2006). A training implication of our findings is the importance of dedicated case‐based learning.…”

Section: Discussionmentioning

confidence: 99%

“…The prevailing empirical literature on mediation is extremely diverse, focusing on such things as the variables that affect the likelihood that mediation will occur, the effectiveness of mediation in different settings, the strategies and tactics used by mediators and the relationship of mediator behavior to outcomes (Kressel, 2006; Wall, Stark, & Standifer, 2001). This body of research has greatly expanded our understanding of the mediation process, but, with some notable exceptions, it largely neglects how experienced mediators think, particularly in terms of global, “top down” cognition that appears characteristic of expert performance in many other domains (Feltovich, Prietula, & Ericsson, 2006; Ross, Schaffer, & Klein, 2006) and that presumably plays an important role in influencing mediator behavior.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Mediating Among Scientists: A Mental Model of Expert Practice

Kressel

Gadlin

2009

Negotiation Conflict Man Res

View full text Add to dashboard Cite

Despite the considerable research on mediator behavior, the cognitive structures and processes that presumably guide the strategic and tactical choices of professional mediators are poorly understood. The current study made use of a reflective case study method to explore in considerable detail the strategic thinking of five experienced mediators. The project was conducted at the National Institutes of Health whose Office of the Ombudsman (OO) mediates disputes among the institute’s scientists. Eighteen cases were studied. The thinking of the mediators in these cases displayed regularities that are described in terms of the ombuds team’s working mental model of mediation. The mental model consists of two strongly contrasting intervention scripts: a deep problem‐solving script (DPS) focused on identifying and addressing latent issues of an interpersonal or systemic kind and a tactical problem‐solving script (TPS) focusing instead on the issues as presented by the parties. The tactical script was applied in either an integrative bargaining mode or a more distributive quasi‐arbitration approach. The choice of which script to follow in a given case is determined by first order decision rules concerning the existence and nature of any latent problems that may be present, and second order decision rules concerning the parties’ capacity to engage in “deep” problem‐solving. Despite their very different foci, both DPS and TPS appear to follow the same metascript of problem‐solving stages, beginning with an intensive diagnostic phase during which the decision rules are applied and a script “selection” is made. DPS is the preferred intervention mode of team members. Every case began with at least a preliminary effort to search for and address latent causes, and team members expressed dissatisfaction if they could not apply DPS in cases where latent problems were thought to be fueling the conflict. However, ombudsmen used the scripts flexibly and switched to TPS if DPS was unnecessary or impractical. Both scripts produced agreements that were useful to the parties and to the institution’s scientific purposes, particularly the fostering of scientific competence. The mental model is heavily shaped by the social context in which the ombudsmen function. Thus, the primacy of DPS in the model appears to be due to the fact that the ombudsmen are “repeat players” in the life of the NIH and therefore in a position to become adept at recognizing the latent sources of its dysfunctional conflicts, are under a strong role mandate as ombudsmen to pay attention to covert patterns of organizational dysfunction, and deal with disputants pressed to address latent issues blocking scientific work.

show abstract

“…The RPD model has been applied to a variety of domains and has received much empirical support, in that experts are found to use recognitional strategies in 80-90% of the cases (see Klein, 1993;Ross, Shafer, & Klein, 2006). One of the intriguing predictions of the RPD model is that the first option considered is usually the best, at least if the professionals making the decision can draw upon extensive domain knowledge and experience.…”

Section: )mentioning

confidence: 99%

Naturalistic Decision Making

Schraagen¹

2014

View full text Add to dashboard Cite

Naturalistic Decision Making (NDM) has been variously described as a 'movement' (Klein, 2015), a 'research community' (Gore, Flin, Stanton, & Wong, 2015), a 'framework' (Klein, 2015, Lipshitz, Klein, Orasanu, & Salas, 2001, and a 'perspective' or 'paradigm ' (Cannon-Bowers, Salas, & Pruitt, 1996). All of these labels are appropriate. As a movement, it originated in 1989 at a small, invitation-only conference in Dayton, Ohio, just one year after the shootdown of an Iranian commercial airliner by a US Navy cruiser, the USS Vincennes. The researchers invited at the 1989 meeting were concerned about applying what was known from the then-existing research on decision making to applied, realworld, contexts, such as the Vincennes tragedy. Their perception of the state of the art of decision making research at that time was that it mainly consisted of laboratory research in which novel tasks were used with inexperienced decision makers (mostly students) who were asked to make a choice among concurrently available alternatives. The findings of this body of research did not generalize to experienced decision makers who often had to make sense of a complex situation before committing themselves to a particular course of action. Thus, a movement was started that evolved into a research community that convened during biennial conferences alternating between the US and Europe. As a movement, then, NDM consists of applied researchers who are interested in how professionals make decisions in real-world situations, with the goal of supporting these professionals through decision aiding and training. The word 'naturalistic' in NDM therefore refers to real-world situations, as contrasted with laboratory situations, rather than 'natural situations' in the sense of 'taking place in nature'.As a framework or perspective, NDM is frequently contrasted with Classical Decision Making (CDM). CDM presents a view of human decision making as fundamentally flawed compared to a normative model. The normative model describes decision making as an exhaustive comparison of options, based on all available information about the options, their weights and consequences. NDM as a perspective on decision making emphasizes the study of how people use their experience to actually make decisions in field settings, rather than how they are supposed to make decisions. In the NDM framework, professional decision making behavior is an adaptation to uncertain, dynamic, environments, shifting, ill-defined or competing goals, time stress, high stakes, multiple eventfeedback loops, ill-structured problems, multiple players, and organizational goals and norms that must be aligned with the decision maker's personal goals and norms (Orasanu & Connolly, 1993). Given these task constraints, decision making does not usually allow for an exhaustive comparison of options, as CDM would claim. The adaptations to these task constraints are usually viewed as successful (Kahneman & Klein, 2009), as long as experts can bring to bear their knowledge and experience in order to m...

show abstract

Professional Judgments and “Naturalistic Decision Making”

Cited by 77 publications

References 33 publications

Workplace-based assessment: raters’ performance theories and constructs

Workplace-based assessment: raters’ performance theories and constructs

Mediating Among Scientists: A Mental Model of Expert Practice

Naturalistic Decision Making

Contact Info

Product

Resources

About