Using Situation Awareness and Workload to Predict Performance in Submarine Track Management: A Multilevel Approach

Loft, Shayne; Jooste, Lisa; Li, Yanqi Ryan; Ballard, Timothy; Huf, Samuel; Lipp, Ottmar V.; Visser, Troy A. W.

doi:10.1177/0018720818784803

Cited by 12 publications

(12 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In one counter-intuitive finding, Loft et al (2018) found that while SAGAT was predictive of performance in a submarine management study, this was only at the between subjects level, but not at the within subjects level. This would indicate that the differences in SA between subjects correlated with performance, but not changes in an individual’s SA within a trial, which is hard to explain on any theoretical level.…”

Section: Resultsmentioning

confidence: 98%

“…Previous research has shown that air traffic controllers do not remember aircraft call signs, for example, but rather rely on spatial memory of where air traffic is located (Endsley & Rodgers, 1998). By pinning each query to the ship contact number in the Loft et al (2018) study, rather than a spatial map as is typical with SAGAT administration, it is possible that the researchers created an artificial memory requirement that interfered with their results. This matter will require further research.…”

Section: Resultsmentioning

confidence: 99%

“…This is because identifiers (e.g., aircraft call signs, track numbers, train numbers) are often not known, but rather are peripheral information that serve a functional task, but have little value otherwise in terms of situation understanding (Endsley, 2015a; Endsley & Rodgers, 1998). Implementations of SAGAT measures that rely heavily on such identifiers to answer queries create an artificial memory requirement that is likely to lead to difficulties such as experienced by Golightly, Wilson, Lowe, and Sharples (2010) in train operations, or Loft et al (2018) in submarine management. Also Ratwani and Trafton (2008) found that spatial memory (such as that provided with maps in SAGAT) is important for resumption from an interruption.…”

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

2019

View full text Add to dashboard Cite

Objective: To examine evidence of sensitivity, predictiveness, and methodological concerns regarding direct, objective measures of situation awareness (SA). Background: The ability to objectively measure SA is important to the evaluation of user interfaces and displays, training programs, and automation initiatives, as well as for studies that seek to better understand SA in both individuals and teams. A number of methodological criticisms have been raised creating significant confusion in the research field. Method: A meta-analysis of 243 studies was conducted to examine evidence of sensitivity and predictiveness, and to address methodological questions regarding Situation Awareness Global Assessment Technique (SAGAT), Situation Present Assessment Technique (SPAM), and their variants. Results: SAGAT and SPAM were found to be equally predictive of performance. SPAM (64%) and real-time probes (73%) were found to have significantly lower sensitivity in comparison to SAGAT (94%). While SAGAT was found not to be overly memory reliant nor intrusive into operator performance, SPAM resulted in problems with intrusiveness in 40% of the studies examined, as well as problems with speed-accuracy tradeoffs, sampling bias, and confounds with workload. Concerns about memory reliance, the utility of these measures for assessing Team SA, and other issues are also addressed. Conclusion: SAGAT was found to be a highly sensitive, reliable, and predictive measure of SA that is useful across a wide variety of domains and experimental settings. Application: Direct, objective SA measurement provides useful and diagnostic insights for research and design in a wide variety of domains and study objectives.

show abstract

Section: Resultsmentioning

confidence: 98%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

2019

View full text Add to dashboard Cite

show abstract

“…Surprisingly, only two studies have examined the relationship between workload and performance in real-time (as opposed to aggregated at a manipulation level, such as easy vs. difficult; Loft et al, 2018;Mracek, Arsenault, Day, Hardy, & Terry, 2014). Both showed that increased workload for an individual (i.e., increased workload relative to one's own average workload) was associated with a decrease in subsequent performance (i.e., decreased performance relative to one's average performance) in command and control tasks.…”

Section: Workload and Performancementioning

confidence: 99%

Using Past and Present Indicators of Human Workload to Explain Variance in Human Performance

et al. 2021

Self Cite

View full text Add to dashboard Cite

Cognitive workload is assumed to influence performance due to resource competition. However, there is a lack of evidence for a direct relationship between changes in workload within an individual over time and changes in that individual's performance. We collected performance data using a multiple object-tracking task in which we measured workload objectively in real-time using a modified detection response task. Using a multi-level Bayesian model controlling for task difficulty and past performance, we found strong evidence that workload both during and preceding a tracking trial was predictive of performance, such that higher workload led to poorer performance. These negative workload-performance relationships were remarkably consistent across individuals. Importantly, we demonstrate that fluctuations in workload independent from the task demands accounted for significant performance variation. The outcomes have implications for designing real-time adaptive systems to proactively mitigate human performance decrements, but also highlight the pervasive influence of cognitive workload more generally.

show abstract

“…All results we included here evaluated correlations between-participants, either averaged across participants or separate analysis of each SA-performance pairing. A handful of papers did evaluate relationships for SA-performance within-participants, but results were varied (Loft et al 2018;O'Hagan et al 2019;Strybel et al 2009). We were unable to include the small number of within-participant results because the output of multilevel models could not be converted to correlation coefficients or the results were overfit.…”

Section: Between-participants Versus Within-participantsmentioning

confidence: 99%

The validity of situation awareness for performance: a meta-analysis

Bakdash

Marusich

Cox

et al. 2021

Theoretical Issues in Ergonomics Science

View full text Add to dashboard Cite

Major changes in version 4 of the preprint: We added additional information and analyses in response to reviewers.-Item 1: Conclusions too strong? (R1). In the response letter, we describe how metaanalysis typically permits stronger and more generalizable inferences than narrative reviews and individual studies. -Item 2: Value of meta-analysis in general? (R2). Note that Bakdash et al. 2020 is a critique of an existing synthesis, but not a meta-analysis in its own right. The critique points out the need for a correctly-implemented, quantitative meta-analysis based on systematic review, a gap filled by the current work. We also clarify the scientific value of meta-analysis in the response letter (also see Item 1). -Item 3: Clarify prediction interval (R3). We revised the explanation and added equations.-Item 4: Drawn relevant connections (R2 and R3). We have drawn connections in the Discussion to relevant literature/fields suggested by R2 and R3We also had several minor corrections to the data: we corrected the sample sizes in two papers (Albina, 2019 [N = 30 to N = 28]; Onal et al. 2014 [N = 94 to N = 95]), added coding for a second experiment in one paper where all effects were previously coded as a single experiment (Lafond et al. 2012), and excluded one paper that does not appear to have assessed SA-performance associations (Dong et al. 2003). The meta-analysis now has 678 effects from 77 papers.Major changes in version 3 of the preprint: New systematic review from scratch, better detection and exclusion of overfit results (excessive degrees of freedom), meta-analysis now has 682 effects from 78 papers, new draw method for estimating ghost results, and analyses of proportions of effects below three meaningful thresholds. The validity of situation awareness for performance: A meta-analysisSituation awareness (SA) is a widely used cognitive construct in human factors, often theoretically posited to be a critical causal factor and/or construct for performance.However, there are concerns that SA may not sufficiently capture the psychological processes underlying performance. We address these conflicting perspectives using metaanalysis to evaluate the patterns of associations among SA-performance effect sizes.Specifically, we focus on the validity of SA for performance-how well SA captures the relevant psychological processes for task performance. In our systematic review of the empirical literature, we coded associations of ten unique measures of SA with performance: 678 effects from 77 papers. The meta-analytic means for SA measures were all of approximately medium or lower effect sizes. The overall mean effect, while significant, was also limited in magnitude (r = 0.26, p < 0.001). Furthermore, there was high unexplained systematic variation with an enormous plausible range for individual effects (r = -0.15 to 0.60). The results indicate that SA's validity for performance tends to be, on average, weak with large variations among effects. Interventions that improve SA may not correspond to meaningful improvement...

show abstract

Using Situation Awareness and Workload to Predict Performance in Submarine Track Management: A Multilevel Approach

Abstract: Adapting automation is more likely to produce optimal outcomes if based on measurement of operator states that predict future task performance, such as workload.

Cited by 12 publications

References 32 publications

A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

Using Past and Present Indicators of Human Workload to Explain Variance in Human Performance

The validity of situation awareness for performance: a meta-analysis

Contact Info

Product

Resources

About