Automation reliability, human–machine system performance, and operator compliance: A study with airport security screeners supported by automated explosives detection systems for cabin baggage screening

Huegli, David; Merks, Sarah; Schwaninger, Adrian

doi:10.1016/j.apergo.2020.103094

Cited by 30 publications

(13 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, direct cueing systems risk attentional tunneling (Krupinski et al, 1993; Maltz & Shinar, 2003; Yeh et al, 2000), especially in response to unreliable automation cues (Goh et al, 2005). In tests among professional screeners, moreover, direct cueing seems to be beneficial primarily to novice screeners (i.e., <1 year experience; e.g., Hättenschwiler et al, 2018), and when unaided performance is low (Huegli et al, 2020). Research of model operators’ strategies for target detection and localization (Cameron et al, 2004; Palmer et al, 2000) using spatial cues could help designers determine how closely observed performance approaches statistically ideal levels and identify sources of suboptimality.…”

Section: Discussionmentioning

confidence: 99%

Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task

2021

View full text Add to dashboard Cite

Objective The present study replicated and extended prior findings of suboptimal automation use in a signal detection task, benchmarking automation-aided performance to the predictions of several statistical models of collaborative decision making. Background Though automated decision aids can assist human operators to perform complex tasks, operators often use the aids suboptimally, achieving performance lower than statistically ideal. Method Participants performed a simulated security screening task requiring them to judge whether a target (a knife) was present or absent in a series of colored X-ray images of passenger baggage. They completed the task both with and without assistance from a 93%-reliable automated decision aid that provided a binary text diagnosis. A series of three experiments varied task characteristics including the timing of the aid’s judgment relative to the raw stimuli, target certainty, and target prevalence. Results and Conclusion Automation-aided performance fell closest to the predictions of the most suboptimal model under consideration, one which assumes the participant defers to the aid’s diagnosis with a probability of 50%. Performance was similar across experiments. Application Results suggest that human operators’ performance when undertaking a naturalistic search task falls far short of optimal and far lower than prior findings using an abstract signal detection task.

show abstract

Section: Discussionmentioning

confidence: 99%

Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task

2021

View full text Add to dashboard Cite

show abstract

“…Our final objective was to investigate the task features and the context in which the interaction took place. Thus far, the research on trust in automation (Dzindolet et al, 2002; Hoff & Bashir, 2015; Lyons et al, 2018) has mainly involved automated systems which support some sort of (visual) detection task (e.g., luggage screening or specific patterns in x-rays; e.g., Huegli et al, 2020, Rieger & Manzey, 2020; Rieger et al, 2021). These tasks usually have directly quantifiable results and may be perceived to suit the abilities of an automated system better than a human.…”

Section: Task Featuresmentioning

confidence: 99%

How Much Reliability Is Enough? A Context-Specific View on Human Interaction With (Artificial) Agents From Different Perspectives

Appelganc

Rieger

Roesler

et al. 2022

Journal of Cognitive Engineering and Decision Making

View full text Add to dashboard Cite

Tasks classically performed by human–human teams in today’s workplaces are increasingly given to human–technology teams instead. The role of technology is not only played by classic decision support systems (DSSs) but more and more by artificial intelligence (AI). Reliability is a key factor influencing trust in technology. Therefore, we investigated the reliability participants require in order to perceive the support agents (human, AI, and DSS) as “highly reliable.” We then examined how trust differed between these highly reliable agents. Whilst there is a range of research identifying trust as an important determinant in human–DSS interaction, the question is whether these findings are also applicable to the interaction between humans and AI. To study these issues, we conducted an experiment ( N = 300) with two different tasks, usually performed by dyadic teams (loan assignment and x-ray screening), from two different perspectives (i.e., working together or being evaluated by the agent). In contrast to our hypotheses, the required reliability if working together was equal regardless of the agent. Nevertheless, participants trusted the human more than an AI or DSS. They also required that AI be more reliable than a human when used to evaluate themselves, thus illustrating the importance of changing perspective.

show abstract

“…A second factor which might affect visual search performance in luggage screening is automation. In recent years, automated decision support systems (DSSs) have been introduced to support screeners detect potential threat items (e.g., Chavaillaz et al 2018Chavaillaz et al , 2019Hättenschwiler et al 2018;Huegli et al 2020). Based on an automated image analysis, the DSS provides the operator with an automated recommendation about the true state of the world (in the current case, information whether a bag contains a target or not).…”

mentioning

confidence: 99%

Visual search behavior and performance in luggage screening: effects of time pressure, automation aid, and target expectancy

2021

View full text Add to dashboard Cite

Visual inspection of luggage using X-ray technology at airports is a time-sensitive task that is often supported by automated systems to increase performance and reduce workload. The present study evaluated how time pressure and automation support influence visual search behavior and performance in a simulated luggage screening task. Moreover, we also investigated how target expectancy (i.e., targets appearing in a target-often location or not) influenced performance and visual search behavior. We used a paradigm where participants used the mouse to uncover a portion of the screen which allowed us to track how much of the stimulus participants uncovered prior to their decision. Participants were randomly assigned to either a high (5-s time per trial) or a low (10-s time per trial) time-pressure condition. In half of the trials, participants were supported by an automated diagnostic aid (85% reliability) in deciding whether a threat item was present. Moreover, within each half, in target-present trials, targets appeared in a predictable location (i.e., 70% of targets appeared in the same quadrant of the image) to investigate effects of target expectancy. The results revealed better detection performance with low time pressure and faster response times with high time pressure. There was an overall negative effect of automation support because the automation was only moderately reliable. Participants also uncovered a smaller amount of the stimulus under high time pressure in target-absent trials. Target expectancy of target location improved accuracy, speed, and the amount of uncovered space needed for the search.Significance Statement Luggage screening is a safety–critical real-world visual search task which often has to be done under time pressure. The present research found that time pressure compromises performance and increases the risk to miss critical items even with automation support. Moreover, even highly reliable automated support may not improve performance if it does not exceed the manual capabilities of the human screener. Lastly, the present research also showed that heuristic search strategies (e.g., areas where targets appear more often) seem to guide attention also in luggage screening.

show abstract

Automation reliability, human–machine system performance, and operator compliance: A study with airport security screeners supported by automated explosives detection systems for cabin baggage screening

Cited by 30 publications

References 56 publications

Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task

Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task

How Much Reliability Is Enough? A Context-Specific View on Human Interaction With (Artificial) Agents From Different Perspectives

Visual search behavior and performance in luggage screening: effects of time pressure, automation aid, and target expectancy

Contact Info

Product

Resources

About