Sampling error and other statistical problems with query-based situation awareness measures

Landry, Steven J.; Yoo, Hyo Sang

doi:10.1177/1071181312561068

Cited by 1 publication

(1 citation statement)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The SAGAT methodology recommends scoring each query separately because many design interventions can effect changes in just one or two elements of SA, often in different directions, and if the queries are combined these differences can cancel each other out (Endsley, 2000). This is consistent with Landry and Yoo (2012) who recommend scoring by query rather than a combined score to avoid the problem of assumed equal likelihood of each element. Analysis of SAGAT queries has also shown a lack of internal consistency (Endsley, 1990b, 2000), arguing against combining across queries.…”

Section: Resultssupporting

confidence: 85%

A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

2019

View full text Add to dashboard Cite

Objective: To examine evidence of sensitivity, predictiveness, and methodological concerns regarding direct, objective measures of situation awareness (SA). Background: The ability to objectively measure SA is important to the evaluation of user interfaces and displays, training programs, and automation initiatives, as well as for studies that seek to better understand SA in both individuals and teams. A number of methodological criticisms have been raised creating significant confusion in the research field. Method: A meta-analysis of 243 studies was conducted to examine evidence of sensitivity and predictiveness, and to address methodological questions regarding Situation Awareness Global Assessment Technique (SAGAT), Situation Present Assessment Technique (SPAM), and their variants. Results: SAGAT and SPAM were found to be equally predictive of performance. SPAM (64%) and real-time probes (73%) were found to have significantly lower sensitivity in comparison to SAGAT (94%). While SAGAT was found not to be overly memory reliant nor intrusive into operator performance, SPAM resulted in problems with intrusiveness in 40% of the studies examined, as well as problems with speed-accuracy tradeoffs, sampling bias, and confounds with workload. Concerns about memory reliance, the utility of these measures for assessing Team SA, and other issues are also addressed. Conclusion: SAGAT was found to be a highly sensitive, reliable, and predictive measure of SA that is useful across a wide variety of domains and experimental settings. Application: Direct, objective SA measurement provides useful and diagnostic insights for research and design in a wide variety of domains and study objectives.

show abstract