The Harder They Fall? A Response to Wickens et al. (2019) Regarding the Generalizability of Lumberjack Predictions to Complex Work Settings

Abstract

Objective This article is a response to Wickens et al.’s (2019) critique of Jamieson and Skraaning (2019). Background Wickens et al. (2019) offer a five-point critique of Jamieson and Skraaning (2019) that they claim tempers the strength of our conclusions. Approach We first correct a misrepresentation in the critique and then respond to each of the criticisms. Results We preserve the strength of our skeptical conclusions about the applicability of the lumberjack model to complex work settings. Applications We continue to caution system designers about the lack of evidence supporting the lumberjack model in the context of complex work systems.

Overcoming the Lumberjack Effect Through Adaptive Autonomy

“…This research supports the concept of a Lumberjack Effect-the higher the trees, the harder they fall when they encounter an ax (Sebok & Wickens, 2017). This concept and the research that led to its application to DOAs has received some push-back from the Human Factors community concerned over its design implications (Jamieson & Skraaning, 2020a, 2020b. In particular, critics question its ability to be generalized to complex work settings (Jamieson & Skraaning, 2020b).…”

“…The Lumberjack Effect and its design implications has been an area of contention in the Human Factors community for several years (Jamieson & Skraaning, 2020a, 2020bWickens et al, 2010Wickens et al, , 2020. This paper presents the idea that designers do not have to choose between accepting the risk of the fall that the Lumberjack Effect describes and obtaining the performance advantages of higher autonomy levels.…”

“…In order to validate the model and its application to a wide variety of work contexts, future research that assesses the situational awareness and failure performance at different DOAs for multiple contexts should be conducted. Additionally, in recognition of the contention surrounding the existence of the Lumberjack Effect in complex work scenarios (Jamieson & Skraaning, 2020a, 2020b, future research should also focus on to what degree the Lumberjack Effect exists in different contexts. This research is essential for the community to assess in what such contexts the critical point identified in our model exists and if the risk of the Lumberjack Effect is great enough to warrant adaptation to a lower LOA.…”

“…This concept and the research that led to its application to DOAs has received some push-back from the Human Factors community concerned over its design implications (Jamieson & Skraaning, 2020a, 2020b. In particular, critics question its ability to be generalized to complex work settings (Jamieson & Skraaning, 2020b). While this controversy is not the primary purpose of this paper, the concerns over the design implications of accepting the Lumberjack Effect as true and applicable to these situations are a motivating factor behind our model.…”

A New Experimental Paradigm to Manipulate Risk in Human-Automation Research

“…Consequently, it might be less likely for the operator to detect an automation error and take over control if needed during automation breakdowns ( Parasuraman & Manzey, 2010 ). Despite a considerable amount of laboratory-based evidence, the transferability of these results to the real-world work context has been partly challenged ( Jamieson & Skraaning, 2019 , 2020 ; Wickens et al 2020 ). A major point of criticism is the insufficient regard of contextual factors of the operators’ working environment.…”