The Absence of Degree of Automation Trade-Offs in Complex Work Settings

Abstract

Objective The objective of this study was to test the predictions of the routine-failure trade-off (or lumberjack) model in a full-scope simulator study with expert operators performing realistic control tasks. Background A meta-study of degree of automation (DOA) studies concluded that DOA predicts task performance under both routine and automation failure conditions, workload, and situation awareness. Empirical support for this conclusion appears to be weak for complex work situations. Method A full-scope nuclear power plant simulator experiment was conducted in which licensed operating crews completed realistic procedure execution tasks. Dependent measures selected from the lumberjack model were collected and analyzed for systematic effects. Results Situation awareness increased with increasing DOA, which contradicts the lumberjack model. Anticipated workload and failure task performance effects were not observed. Conclusion The experimental results add further evidence challenging the applicability of the lumberjack model to complex work situations. Application Practitioners should use caution when extending the predictions of the lumberjack model based on data from simple work situations to complex work situations. Researchers should invest more resources in testing the predictive power of the lumberjack model in complex work situations.

Absence of DOA Effect but No Proper Test of the Lumberjack Effect: A Reply to Jamieson and Skraaning (2019)

“…Jamieson and Skraaning (2019) have criticized the lumberjack effect of human–automation interaction as not applicable to real world operators in a complex process control simulation, on the basis of their findings that fail to replicate key features of the effect, particularly the loss of SA with a higher DOA.…”

“…Jamieson and Skraaning (2019) now have followed up on this question in carrying out a study which they argue to be a proper test of the lumberjack effect within a highly realistic scenario and employing well-trained professionals. We appreciate this effort as studies under realistic conditions are still an exception in human–automation research and are certainly highly needed.…”

Towards a Deeper Understanding of Automation Transparency in the Operation of Nuclear Plants

“…CBP functions include automatic place keeping, correct component verification, calculations, integration with soft controls, and selective enabling of procedure steps that are relevant to the operating context. Automatic execution of these functions should minimize administrative and operational errors; however, questions remain regarding what the operators need to know with respect to how these dynamic instructions work Skraaning and Jamieson, 2020).…”

The Harder They Fall? A Response to Wickens et al. (2019) Regarding the Generalizability of Lumberjack Predictions to Complex Work Settings

“…It seems likely that this mischaracterization is based on Wickens’s (2018) opinion that accumulated null results serve as neither confirming nor disproving evidence for a posited human performance model. We stated our opposition to this thinking as follows: “If thorough investigations are conducted to observe anticipated effects, and those effects are absent, there are no logical reasons (see Copi, 1953, p. 57) for dismissing the findings as uninformative” (Jamieson & Skraaning, 2019, p. 11). Wickens et al (2019) offer no reasons for dismissing the null findings in our table 3.…”