Designing Human-Autonomy Teaming Experiments Through Reinforcement Learning

Schelble, Beau G.; Canonico, Lorenzo-Barberis; McNeese, Nathan J.; Carroll, J. M.; Hird, Casey

doi:10.1177/1071181320641340

Cited by 8 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current study employs a mixed-methods design to capture and analyze team cognition's formation in teams with varying numbers of agents and humans. The experiment utilized the well-published and validated team research platform known as NeoCITIES [38,39,45,62], which provides an excellent environment to study team cognition and team interaction within both human-human teams [34], and HATs [82]. A 1x3 experimental design was developed as shown in Table 1 to study the effect of various team compositions on the development of team cognition and its related outcomes.…”

Section: Methodsmentioning

confidence: 99%

“…Alternatively, other experiments display incredible HAT performance, outpacing not only human-human teams but even teams consisting of all AI [95]. This disparity can likely be attributed to three major factors: 1) not all studies use true AI, which is capable of expert-level performance when properly trained [25]; 2) the more abstract the team task is, the harder it becomes to train high performing agents [82] (but not impossible [40]); and 3) a distinct lack of proper design and integration within human-agent teams that leads to confusion and poor understanding between the two types of team members. The solution to this discrepancy is to leverage the individual strengths of the AI and the human to move team effectiveness beyond what each is capable of achieving alone.…”

Section: Human-agent Teams Performance and Trustmentioning

confidence: 99%

See 1 more Smart Citation

Let's Think Together! Assessing Shared Mental Models, Performance, and Trust in Human-Agent Teams

Schelble

Flathmann

McNeese

et al. 2022

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

An emerging research agenda in Computer-Supported Cooperative Work focuses on human-agent teaming and AI agent's roles and effects in modern teamwork. In particular, one understudied key question centers around the construct of team cognition within human-agent teams. This study explores the unique nature of team dynamics in human-agent teams compared to human-human teams and the impact of team composition on perceived team cognition, team performance, and trust. In doing so, a mixed-method approach, including three team composition conditions (all human, human-human-agent, human-agent-agent), completed the team simulation NeoCITIES and completed shared mental model, trust, and perception measures. Results found that human-agent teams are similar to human-only teams in the iterative development of team cognition and the importance of communication to accelerating its development; however, human-agent teams are different in that action-related communication and explicitly shared goals are beneficial to developing team cognition. Additionally, human-agent teams trusted agent teammates less when working with only agents and no other humans, perceived less team cognition with agent teammates than human ones, and had significantly inconsistent levels of team mental model similarity when compared to human-only teams. This study contributes to Computer-Supported Cooperative Work in three significant ways: 1) advancing the existing research on human-agent teaming by shedding light on the relationship between humans and agents operating in collaborative environments, 2) characterizing team cognition development in human-agent teams; and 3) advancing real-world design recommendations that promote human-centered teaming agents and better integrate the two.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Human-agent Teams Performance and Trustmentioning

confidence: 99%

Let's Think Together! Assessing Shared Mental Models, Performance, and Trust in Human-Agent Teams

Schelble

Flathmann

McNeese

et al. 2022

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

show abstract

“…The team simulation used in this study was NeoCITIES, which recently underwent a rigorous redesign that allowed the integration of AI team members [59]. The NeoCITIES experimental platform has a long history in team research and has been used for a variety of previously published team research [60]- [62].…”

Section: B Neocities Task and Rolesmentioning

confidence: 99%

“…The NeoCITIES experimental platform has a long history in team research and has been used for a variety of previously published team research [60]- [62]. The redesign included a map for human participants, a complete UI overhaul (seen in Figure 1), and a back-end architecture redesign that allowed for simultaneous sessions and real-time game state tracking (for more details on the NeoCITIES redesign, please see [59]). The NeoCTIES task simulation requires three separate team members to coordinate and complete a complex task, which develops various constructs essential to effective team outputs like situational awareness [22] and team cognition [14].…”

Section: B Neocities Task and Rolesmentioning

confidence: 99%

Who/What is My Teammate? Team Composition Considerations in Human-AI Teaming

McNeese¹,

Schelble²,

Canonico³

et al. 2021

Preprint

View full text Add to dashboard Cite

There are many unknowns regarding the characteristics and dynamics of human-AI teams, including a lack of understanding of how certain human-human teaming concepts may or may not apply to human-AI teams and how this composition affects team performance. This paper outlines an experimental research study that investigates essential aspects of human-AI teaming such as team performance, team situation awareness, and perceived team cognition in various mixed composition teams (human-only, human-human-AI, human-AI-AI, and AI-only) through a simulated emergency response management scenario. Results indicate dichotomous outcomes regarding perceived team cognition and performance metrics, as perceived team cognition was not predictive of performance. Performance metrics like team situational awareness and team score showed that teams composed of all human participants performed at a lower level than mixed human-AI teams, with the AI-only teams attaining the highest performance. Perceived team cognition was highest in human-only teams, with mixed composition teams reporting perceived team cognition 58% below the all-human teams. These results inform future mixed teams of the potential performance gains in utilizing mixed teams' over human-only teams in certain applications, while also highlighting mixed teams' adverse effects on perceived team cognition.

show abstract

“…Step 3 -Development of Test Platform. Many HAT research studies have been conducted using Wizard of Oz technique which mimics the autonomous systems simulated by either a human or a scripted scenario with potential event trees (Schelble, Canonico, McNeese, Carroll, & Hird, 2020). This technique enables testing various assumptions using system prototypes with various fidelity levels.…”

Section: Human Factors Guidance For Hat Developmentmentioning

confidence: 99%

Developing human-autonomy teaming characteristics and guidance in aerospace operations

Tokadli

View full text Add to dashboard Cite

show abstract

Designing Human-Autonomy Teaming Experiments Through Reinforcement Learning

Cited by 8 publications

References 16 publications

Let's Think Together! Assessing Shared Mental Models, Performance, and Trust in Human-Agent Teams

Let's Think Together! Assessing Shared Mental Models, Performance, and Trust in Human-Agent Teams

Who/What is My Teammate? Team Composition Considerations in Human-AI Teaming

Developing human-autonomy teaming characteristics and guidance in aerospace operations

Contact Info

Product

Resources

About