HINT: Integration Testing for AI-based features with Humans in the Loop

Chen, Quan Ze; Schnabel, Tobias; Nushi, Besmira; Amershi, Saleema

doi:10.1145/3490099.3511141

Cited by 10 publications

(4 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cruz et al (2021) specifically used the term "human-robot interaction" rather than teaming, where the robot provides explanations of its actions to a human who is not directly involved in the task. Emphasizing the "dynamic experience" (Chen et al, 2022, p. 549) of both parties adapting to each other, Chen et al (2022) used mostly the term "human-AI collaboration". They adopted a human-centered perspective on AI and the development of collaboration.…”

Section: Understandings Of Human-ai Teamingmentioning

confidence: 99%

Defining human-AI teaming the human-centered way: a scoping review and network analysis

Berretta,

Tausch,

Ontrup

et al. 2023

Front. Artif. Intell.

View full text Add to dashboard Cite

IntroductionWith the advancement of technology and the increasing utilization of AI, the nature of human work is evolving, requiring individuals to collaborate not only with other humans but also with AI technologies to accomplish complex goals. This requires a shift in perspective from technology-driven questions to a human-centered research and design agenda putting people and evolving teams in the center of attention. A socio-technical approach is needed to view AI as more than just a technological tool, but as a team member, leading to the emergence of human-AI teaming (HAIT). In this new form of work, humans and AI synergistically combine their respective capabilities to accomplish shared goals.MethodsThe aim of our work is to uncover current research streams on HAIT and derive a unified understanding of the construct through a bibliometric network analysis, a scoping review and synthetization of a definition from a socio-technical point of view. In addition, antecedents and outcomes examined in the literature are extracted to guide future research in this field.ResultsThrough network analysis, five clusters with different research focuses on HAIT were identified. These clusters revolve around (1) human and (2) task-dependent variables, (3) AI explainability, (4) AI-driven robotic systems, and (5) the effects of AI performance on human perception. Despite these diverse research focuses, the current body of literature is predominantly driven by a technology-centric and engineering perspective, with no consistent definition or terminology of HAIT emerging to date.DiscussionWe propose a unifying definition combining a human-centered and team-oriented perspective as well as summarize what is still needed in future research regarding HAIT. Thus, this work contributes to support the idea of the Frontiers Research Topic of a theoretical and conceptual basis for human work with AI systems.

show abstract

Section: Understandings Of Human-ai Teamingmentioning

confidence: 99%

Defining human-AI teaming the human-centered way: a scoping review and network analysis

Berretta,

Tausch,

Ontrup

et al. 2023

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…In addition, human usage of AI tools is naturally highly varied; it may not be possible to optimise a single model for all behaviour patterns, especially in cases where those behaviours are contradictory in nature. Novel evaluation frameworks are therefore required to assess human behaviour patterns with AI tools [35], and establish what affects these trends have on the trustworthy characteristics described above.…”

Section: Figure 2 Components Within the Uk National Quality Infrastru...mentioning

confidence: 99%

Certification of machine learning applications in the context of trustworthy AI with reference to the standardisation of AI systems

Levene¹,

Wooldridge²

2023

View full text Add to dashboard Cite

Artificial intelligence (AI) and its subset machine learning (ML), which focuses on learning tasks from data, are some of the most disruptive emergent technologies. AI standards inform organisations how to develop and manage their AI systems and are emerging to satisfy the increasing demand from industry and society for the safe adoption of AI and ML technologies. However, AI systems must be trustworthy in the sense that they can be relied upon to make responsible decisions. Consequently, trustworthy AI is a collection of principles that encourages responsible development, use and deployment of AI systems, and can be viewed as a framework for managing risk in AI systems.

show abstract

“…To overcome the challenges of testing human-AI interactions and collaborations, the authors of [37] propose a Human-AI INtergarion Testing framework (HINT). HINT is a crowdbased framework that uses a humans-in-the-loop workflow to test AI-based experiences.…”

Section: Software Verification and Validationmentioning

confidence: 99%

A Review of Trending Crowdsourcing Topics in Software Engineering Highlighting Mobile Crowdsourcing and AI Utilization

Alghasham¹,

Alzakan²,

Al-Hagery³

2023

IJACSA

View full text Add to dashboard Cite

Today's modern technologies and requirements make the utilization of crowdsourcing more viable and applicable. It is one of the problem-solving models that can be used in various domains to reduce costs and time. It is also an excellent way to find new and different ideas and solutions. This paper studies the use of crowdsourcing in software engineering and reveals adequate details to highlight its significance. A few recent literature reviews have been published to address specific topics or study general attributes of papers in crowdsourced software engineering. This paper, however, explores all recent publications related to software and crowdsourcing to find the trends and highlight mobile and AI usage in software crowdsourcing. The findings of this paper show that most research papers are in the areas of software management and software verification and validation. The results also reveal that machine learning and data mining techniques are predominant in software management crowdsourcing and software verification and validation. Furthermore, this study shows that the methods and techniques used in general crowdsourcing apply to mobile crowdsourcing except in mobile testing, where there is a need for clustering and prioritization of test reports.

show abstract

HINT: Integration Testing for AI-based features with Humans in the Loop

Cited by 10 publications

References 43 publications

Defining human-AI teaming the human-centered way: a scoping review and network analysis

Defining human-AI teaming the human-centered way: a scoping review and network analysis

Certification of machine learning applications in the context of trustworthy AI with reference to the standardisation of AI systems

A Review of Trending Crowdsourcing Topics in Software Engineering Highlighting Mobile Crowdsourcing and AI Utilization

Contact Info

Product

Resources

About