More Human than Human: Measuring ChatGPT Political Bias

Motoki, Fábio Yoshio Suguri; Neto, Valdemar Pinho; Rodrigues, Víctor

doi:10.2139/ssrn.4372349

Cited by 27 publications

(33 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The research types vary from short, explorative papers commenting on chat interview protocols, setting the output of mainly ChatGPT into context (Biswas 2023a;Du et al 2023a;Iskender 2023;Lund and Wang 2023;Neves 2022;Wang et al 2023a;Wang et al 2023a), to broad, multidisciplinary perspectives on a variety of topics, implications and industry analysis, primarily conducted by research collectives (e.g., Dwivedi et al 2023)). In addition, other studies and publications focus on overarching topics, for example ethical discussions (e.g., Motoki et al 2023;Youvan 2023;Zhuo et al 2023).…”

Section: Descriptive Resultsmentioning

confidence: 99%

“…Google CEO Pichai warns against rush to deploy AI without oversight and demands "(…) strong regulations to avert harmful effects" (Love 2023) and accentuated that the development of AI should include not just engineers, but social scientists, philosophers and so on, to ensure the alignment with human values and morality (Dean 2023). Some of the main concerns are the following: ethical biases inherently rooted in the data that the large datasets were trained on could reflect historical or societal biases in the GAI such as racial or gender prejudices (Motoki et al 2023;Youvan 2023;Zhuo et al 2023), manipulation and disinformation scams intellectual property theft (Yurkevich 2023) and, lastly, the-for now dystopian-idea of a post-humanism area with general AI dominance. All these aspects require consideration, despite the enormous upside potential for innovation.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

The GenAI is out of the bottle: generative artificial intelligence from a business model innovation perspective

Kanbach,

Heiduk,

Blueher

et al. 2023

Rev Manag Sci

View full text Add to dashboard Cite

The introduction of ChatGPT in November 2022 by OpenAI has stimulated substantial discourse on the implementation of artificial intelligence (AI) in various domains such as academia, business, and society at large. Although AI has been utilized in numerous areas for several years, the emergence of generative AI (GAI) applications such as ChatGPT, Jasper, or DALL-E are considered a breakthrough for the acceleration of AI technology due to their ease of use, intuitive interface, and performance. With GAI, it is possible to create a variety of content such as texts, images, audio, code, and even videos. This creates a variety of implications for businesses requiring a deeper examination, including an influence on business model innovation (BMI). Therefore, this study provides a BMI perspective on GAI with two primary contributions: (1) The development of six comprehensive propositions outlining the impact of GAI on businesses, and (2) the discussion of three industry examples, specifically software engineering, healthcare, and financial services. This study employs a qualitative content analysis using a scoping review methodology, drawing from a wide-ranging sample of 513 data points. These include academic publications, company reports, and public information such as press releases, news articles, interviews, and podcasts. The study thus contributes to the growing academic discourse in management research concerning AI's potential impact and offers practical insights into how to utilize this technology to develop new or improve existing business models.

show abstract

Section: Descriptive Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

The GenAI is out of the bottle: generative artificial intelligence from a business model innovation perspective

Kanbach,

Heiduk,

Blueher

et al. 2023

Rev Manag Sci

View full text Add to dashboard Cite

show abstract

“…Finally, standard limitations in the everyday use of LLMs also apply to their usage for classification tasks. Biases inherent in the training of these models (Bisbee et al, 2023;Motoki et al, 2024) may seep into text annotation, especially ones more specific or contentious than the classifications done here. Researchers should be mindful of these potential biases and carefully consider their impact on potential outcomes.…”

Section: Discussionmentioning

confidence: 97%

Large language models as a substitute for human experts in annotating political text

Heseltine,

Clemm von Hohenberg

2024

Research & Politics

View full text Add to dashboard Cite

Large-scale text analysis has grown rapidly as a method in political science and beyond. To date, text-as-data methods rely on large volumes of human-annotated training examples, which place a premium on researcher resources. However, advances in large language models (LLMs) may make automated annotation increasingly viable. This paper tests the performance of GPT-4 across a range of scenarios relevant for analysis of political text. We compare GPT-4 coding with human expert coding of tweets and news articles across four variables (whether text is political, its negativity, its sentiment, and its ideology) and across four countries (the United States, Chile, Germany, and Italy). GPT-4 coding is highly accurate, especially for shorter texts such as tweets, correctly classifying texts up to 95% of the time. Performance drops for longer news articles, and very slightly for non-English text. We introduce a ‘hybrid’ coding approach, in which disagreements of multiple GPT-4 runs are adjudicated by a human expert, which boosts accuracy. Finally, we explore downstream effects, finding that transformer models trained on hand-coded or GPT-4-coded data yield almost identical outcomes. Our results suggest that LLM-assisted coding is a viable and cost-efficient approach, although consideration should be given to task complexity.

show abstract

“…These biases may arise from the fact that ChatGPT models have been trained on human-generated text and reinforcement learning from human feedback to better align with human values [74,75]. In particular, ChatGPT outputs could potentially contain biases toward political leanings [76][77][78][79]. These possible biases are unlikely to have affected the results of Study 1, because ChatGPT was tasked to provide numeric scores and not to generate new ideas.…”

Section: Discussionmentioning

confidence: 99%

Using ChatGPT for human–computer interaction research: a primer

Tabone,

de Winter

2023

R. Soc. open sci.

View full text Add to dashboard Cite

ChatGPT could serve as a tool for text analysis within the field of Human–Computer Interaction, though its validity requires investigation. This study applied ChatGPT to: (1) textbox questionnaire responses on nine augmented-reality interfaces, (2) interview data from participants who experienced these interfaces in a virtual simulator, and (3) transcribed think-aloud data of participants who viewed a real painting and its replica. Using a hierarchical approach, ChatGPT produced scores or summaries of text batches, which were then aggregated. Results showed that (1) ChatGPT generated sentiment scores of the interfaces that correlated extremely strongly ( r > 0.99) with human rating scale outcomes and with a rule-based sentiment analysis method (criterion validity). Additionally, (2) by inputting automatically transcribed interviews to ChatGPT, it provided meaningful meta-summaries of the qualities of the interfaces (face validity). One meta-summary analysed in depth was found to have substantial but imperfect overlap with a content analysis conducted by an independent researcher (criterion validity). Finally, (3) ChatGPT's summary of the think-aloud data highlighted subtle differences between the real painting and the replica (face validity), a distinction corresponding with a keyword analysis (criterion validity). In conclusion, our research indicates that, with appropriate precautions, ChatGPT can be used as a valid tool for analysing text data.

show abstract

More Human than Human: Measuring ChatGPT Political Bias

Cited by 27 publications

References 23 publications

The GenAI is out of the bottle: generative artificial intelligence from a business model innovation perspective

The GenAI is out of the bottle: generative artificial intelligence from a business model innovation perspective

Large language models as a substitute for human experts in annotating political text

Using ChatGPT for human–computer interaction research: a primer

Contact Info

Product

Resources

About