A review of possible effects of cognitive biases on interpretation of rule-based machine learning models

Kliegr, Tomáš; Bahník, Štěpán; Fürnkranz, Johannes

doi:10.1016/j.artint.2021.103458

Cited by 72 publications

(37 citation statements)

References 126 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main findings from our quantitative studies (see Table 5 for a schematic summary), confirm the literature on the biases implied by agential AI in decision support [11,22,33,74] (case 1) and also suggest that group discussion can leverage collective intelligence and lead to a better performance than superhuman AI (case 2). The two qualitative case studies, on the other hand, shed light on the need to contextualize (in the broadest sense) the AI advice, so as to support the users' creative appropriation of the technology.…”

Section: The Concepts Of Knowledge Artifact and Basupporting

confidence: 82%

“…Nonetheless, recent research [37,59,70] has highlighted how also such methods can lead to problematic appropriation, primarily due to the emergence of biases that may affect the interaction between the human users and the AI systems and that can be reinforced by XAI methods [33], e.g. automation bias and the "white box paradox" [6].…”

Section: Background Motivations and Related Workmentioning

confidence: 99%

“…However, also these latter approaches have not focused much on the potentially harmful effects of AI support on the cognition of human individuals who interact with these systems [33,59,70]. These effects include automation bias [22] (i.e., the propensity of humans to favor suggestions from AI systems); automation complacency [4] (i.e., the unjustified satisfaction with an AI system, possibly leading to non-vigilance, or poor detection of malfunctions); and "prejudices against the machine" [11,74] (i.e., the opposite of automation bias, the propensity to reject the suggestion from AI systems).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The need to move away from agential-AI: Empirical investigations, useful concepts and open issues

Cabitza

Campagner

Simone

2021

International Journal of Human-Computer Studies

View full text Add to dashboard Cite

We propose a novel approach to human interaction with artificial intelligence systems (HAII), alternative to the mainstream dyadic one where humans and AI are seen as interacting agents. Through two quantitative experiments and two qualitative in-field case studies, we show that the mainstream HAII paradigm presents potentially harmful design shortcomings as it can trigger negative dynamics such as automation bias and prejudices. Our proposal, on the other hand, is grounded in the Computer-Supported Cooperative Work literature, in which AI can be conceived as a component of a Knowledge Artifact (KA). This consists of an ecosystem of knowledge creation tools whose goal is to support a Ba (after Nonaka), i.e. a collective of competent decision makers. We highlight the cooperative nature of decision making and the AI functionalities that a KA should embed. These include eXplainable AI solutions, aimed at facilitating appropriation, but also functionalities that enable reasoning in a collaborative setting. Finally, we discuss how moving intelligence and agency from individual agents to the human collective can help to mitigate the shortcomings of dyadic HAII (e.g., deskilling), re-distribute responsibility in critical tasks, and revisit the HAII research agenda to align it with the needs of increasingly wide, heterogeneous and complex teams.

show abstract

Section: The Concepts Of Knowledge Artifact and Basupporting

confidence: 82%

Section: Background Motivations and Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The need to move away from agential-AI: Empirical investigations, useful concepts and open issues

Cabitza

Campagner

Simone

2021

International Journal of Human-Computer Studies

View full text Add to dashboard Cite

show abstract

“…Across these cases biases can creep into the design and outputs (Blodgett et al, 2020). Supervised ML can reflect the cognitive biases of the designers, hidden in the logical decision-making rules of the algorithm in producing interpretable results (Kliegr et al, 2021). Unsupervised ML can acquire stereotyped biases from textual data that reflect human culture (Caliskan et al, 2017).…”

Section: Ai and Ml: Emerging Ethical Challengesmentioning

confidence: 99%

Can AI be racist? Color‐evasiveness in the application of machine learning to science assessments

Cheuk

2021

Science Education

View full text Add to dashboard Cite

Assessment developers are increasingly using the developing technology of machine learning in transforming how to assess students in their science learning. I argue that these algorithmic models further embed the structures of inequality that are pervasive in the development of science assessments in how they legitimize certain language practices that protect the hierarchical standing of status quo interests. My argument is situated within the broader emerging ethical challenges around this new technology.I apply a raciolinguistic equity analysis framework in critiquing the "new black box" that reinforces structural forms of discrimination against the linguistic repertoires of racially marginalized student populations. The article ends with me sharing a set of tactical shifts that can be deployed to form a more equitable and socially-just field of machine learning enhanced science assessments.

show abstract

“…It is not new for modeling experts to face questions about model acceptability and validation as we expose here, but they were trained in a different context and environment when comparing with the currently era of data scientists exposed to AI. Efforts are being made to promote greater transparency in algorithms with low levels of expert intervention [84][85][86][87][88]. Miller [89] names these efforts as "explainable artificial intelligence research" and considers that there will be an increasing need to integrate AI with other fields of knowledge, such as philosophy, cognitive psychology/science, and social psychology.…”

Section: Decision Makingmentioning

confidence: 99%

New dilemmas, old problems: advances in data analysis and its geoethical implications in groundwater management

Silva¹,

Matulovic

Manzione

2021

SN Appl. Sci.

View full text Add to dashboard Cite

Groundwater governance uses modeling to support decision making. Therefore, data science techniques are essential. Specific difficulties arise because variables must be used that cannot be directly measured, such as aquifer recharge and groundwater flow. However, such techniques involve dealing with (often not very explicitly stated) ethical questions. To support groundwater governance, these ethical questions cannot be solved straightforward. In this study, we propose an approach called “open-minded roadmap” to guide data analytics and modeling for groundwater governance decision making. To frame the ethical questions, we use the concept of geoethical thinking, a method to combine geoscience-expertise and societal responsibility of the geoscientist. We present a case study in groundwater monitoring modeling experiment using data analytics methods in southeast Brazil. A model based on fuzzy logic (with high expert intervention) and three data-driven models (with low expert intervention) are tested and evaluated for aquifer recharge in watersheds. The roadmap approach consists of three issues: (a) data acquisition, (b) modeling and (c) the open-minded (geo)ethical attitude. The level of expert intervention in the modeling stage and model validation are discussed. A search for gaps in the model use is made, anticipating issues through the development of application scenarios, to reach a final decision. When the model is validated in one watershed and then extrapolated to neighboring watersheds, we found large asymmetries in the recharge estimatives. Hence, we can show that more information (data, expertise etc.) is needed to improve the models’ predictability-skill. In the resulting iterative approach, new questions will arise (as new information comes available), and therefore, steady recourse to the open-minded roadmap is recommended. Graphic abstract

show abstract

A review of possible effects of cognitive biases on interpretation of rule-based machine learning models

Cited by 72 publications

References 126 publications

The need to move away from agential-AI: Empirical investigations, useful concepts and open issues

The need to move away from agential-AI: Empirical investigations, useful concepts and open issues

Can AI be racist? Color‐evasiveness in the application of machine learning to science assessments

New dilemmas, old problems: advances in data analysis and its geoethical implications in groundwater management

Contact Info

Product

Resources

About