Persuasive Technology and computational manipulation: hypernudging out of mental self-determination

Abstract

Artificial Intelligence, unperceived, can acquire the user's data, find connections not visible by a human being, profile the users, and aim at persuading them, resulting in Persuasive Technology (PT). During the persuasive process, PT can use manipulation, finding and using routes to affect System 1, the primordial brain of individuals, in the absence of their awareness, undermining their decision-making processes. Multiple international and European bodies recognized that AI systems could use manipulation at an unprecedented degree via second-generation dark patterns such as the hypernudge and that computational manipulation constitutes a risk for autonomy and different, overlapping, fundamental rights such as privacy, informational self-determination and freedom of thought. However, there is a lack of shared ideas regarding which fundamental rights are violated by computational manipulation and which fundamental rights can protect individuals against it. The right to be let alone and the right to hold and express a thought differ from the right to create a thought, being in control of the decision-making process and free from cognitive interferences operated by computational manipulation. Therefore, this paper argues in favor of recognizing a newly emerged fundamental right, the right to mental self-determination, tailored to the unprecedented abilities of AI-driven manipulative technologies.

Ethics of Generative AI and Manipulation: A Design-Oriented Research Agenda

“…in the EU's forthcoming AI Act (European Commission, 2021; European Commission et al, 2022). More generally, manipulation is considered a threat to democracy and trustworthiness, which means that it is a fundamental threat to the critical aim of responsible, trustworthy AI (Faraoni, 2023). In addition, a large body of literature documents worries about manipulation in other contexts, notably nudging and advertising (cf.…”