Harms from Increasingly Agentic Algorithmic Systems

Chan, Alan H. S.; Salganik, Rebecca; Markelius, Alva; Pang, Chris; Rajkumar, Nitarshan; Dmitrii, Krasheninnikov,; Lauro, Langosco,; He, Zhonghao; Duan, Yingli; Carroll, Micah; Lin, Michelle; Mayhew, Alex; Collins, Katherine A.; Molamohammadi, Maryam; Burden, John; Zhao, Wanru; Rismani, Shalaleh; Voudouris, Konstantinos; Bhatt, Umang; Weller, Adrian; Krueger, David A.; Maharaj, Tegan

doi:10.1145/3593013.3594033

Cited by 18 publications

(10 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both because those systems might have moral status, so that we might harm them, and because they might exhibit power-seeking tendencies, so that they might harm us. At the same time, there are strong financial and scientific incentives to build systems which are increasingly agentic (Chan et al 2023). This is worrisome.…”

Section: Discussionmentioning

confidence: 99%

“…The third dimension is efficacy (Chan et al 2023). A being is efficacious if its goal-directed behavior affects the world, without someone else's mediation or intervention.…”

Section: Efficacymentioning

confidence: 99%

“…The fourth dimension is planning. Following Chan et al (2023), I will understand the planning dimension as 'the degree to which the algorithmic system is [able] to make decisions that are temporally dependent upon one another to achieve a goal and/or make predictions over a long time horizon'. That is, the longer a system can plan in advance, the higher it scores on the planning dimension.…”

Section: Planningmentioning

confidence: 99%

See 2 more Smart Citations

Understanding Artificial Agency

Dung

2024

The Philosophical Quarterly

View full text Add to dashboard Cite

Which artificial intelligence (AI) systems are agents? To answer this question, I propose a multidimensional account of agency. According to this account, a system's agency profile is jointly determined by its level of goal-directedness and autonomy as well as is abilities for directly impacting the surrounding world, long-term planning and acting for reasons. Rooted in extant theories of agency, this account enables fine-grained, nuanced comparative characterizations of artificial agency. I show that this account has multiple important virtues and is more informative than alternatives. More speculatively, it may help to illuminate two important emerging questions in AI ethics: 1. Can agency contribute to the moral status of non-human beings, and how? 2. When and why might AI systems exhibit power-seeking behaviour and does this pose an existential risk to humanity?

show abstract

Section: Discussionmentioning

confidence: 99%

“…The third dimension is efficacy (Chan et al 2023). A being is efficacious if its goal-directed behavior affects the world, without someone else's mediation or intervention.…”

Section: Efficacymentioning

confidence: 99%

Section: Planningmentioning

confidence: 99%

See 1 more Smart Citation

Understanding Artificial Agency

Dung

2024

The Philosophical Quarterly

View full text Add to dashboard Cite

show abstract

“…While it is commonplace in both technical and ethical communities to use the latter terminology -for example, referring to conversational agents or dialogue agents -this risks implicitly attributing agential capacities to AI systems that lack them [8]. Clarity on this point is likely to be of growing importance given the increasing agential capabilities of some frontier models [9]. disappear in that period, progress was relatively slow, and was frequently driven by more theoretical interests such as measurement of AI progress rather than direct commercial applications.…”

Section: A Recent History Of Social Aimentioning

confidence: 99%

All too human? Identifying and mitigating ethical risks of Social AI

Shevlin

2024

Law, Ethics &Amp; Technology

View full text Add to dashboard Cite

This paper presents an overview of the risks and benefits of Social AI, understood as conversational AI systems that cater to human social needs like romance, companionship, or entertainment. Section 1 of the paper provides a brief history of conversational AI systems and introduces conceptual distinctions to help distinguish varieties of Social AI and pathways to their deployment. Section 2 of the paper adds further context via a brief discussion of anthropomorphism and its relevance to assessment of human-chatbot relationships. Section 3 of the paper provides a survey of potential and in some cases demonstrated harms associated with user interactions with Social AI systems. Finally, Section 4 discusses how the benefits and harms of Social AI can best be addressed, with a primary focus on how frameworks from AI ethics can inform their development.

show abstract

“…there exist no particular threshold that affords a binary classification of whether a system is anthropomorphic or not, instead anthropomorphism exists on a spectrum. At the most basic level, systems are anthropomorphic if they (i) are interactive, (ii) use language, and (iii) take on a role performed by a human (Chan et al, 2023;Reeves and Nass, 1996). While these characteristics are inherent to dialogue systems, not all systems are equally humanlike.…”

Section: Agent Factorsmentioning

confidence: 99%

Mirages. On Anthropomorphism in Dialogue Systems

Abercrombie,

Curry,

Dinkar

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Automated dialogue or conversational systems are anthropomorphised by developers and personified by users. While a degree of anthropomorphism may be inevitable due to the choice of medium, conscious and unconscious design choices can guide users to personify such systems to varying degrees. Encouraging users to relate to automated systems as if they were human can lead to high risk scenarios caused by over-reliance on their outputs. As a result, natural language processing researchers have investigated the factors that induce personification and develop resources to mitigate such effects. However, these efforts are fragmented, and many aspects of anthropomorphism have yet to be explored. In this paper, we discuss the linguistic factors that contribute to the anthropomorphism of dialogue systems and the harms that can arise, including reinforcing gender stereotypes and notions of acceptable language. We recommend that future efforts towards developing dialogue systems take particular care in their design, development, release, and description; and attend to the many linguistic cues that can elicit personification by users. * Equal contribution. † Now at Google DeepMind.

show abstract

Harms from Increasingly Agentic Algorithmic Systems

Cited by 18 publications

References 11 publications

Understanding Artificial Agency

Understanding Artificial Agency

All too human? Identifying and mitigating ethical risks of Social AI

Mirages. On Anthropomorphism in Dialogue Systems

Contact Info

Product

Resources

About