Småprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning

Abstract

Building open-domain conversational systems (or chatbots) that produce convincing responses is a recognized challenge.Recent state-of-theart (SoTA) transformer-based models for the generation of natural language dialogue have demonstrated impressive performance in simulating human-like, single-turn conversations in English. This work investigates, by an empirical study, the potential for transfer learning of such models to Swedish language. DialoGPT, an English language pre-trained model, is adapted by training on three different Swedish language conversational datasets obtained from publicly available sources. Perplexity score (an automated intrinsic language model metric) and surveys by human evaluation were used to assess the performances of the fine-tuned models, with results that indicate that the capacity for transfer learning can be exploited with considerable success. Human evaluators asked to score the simulated dialogue judged over 57% of the chatbot responses to be human-like for the model trained on the largest (Swedish) dataset. We provide the demos and model checkpoints of our English and Swedish chatbots on the HuggingFace platform for public use.

Examining the Potential Usages, Features, and Challenges of Using ChatGPT Technology: A PRISMA-Based Systematic Review

“…GPT models are specifically designed to generate natural language content (Black et al, 2022;Adewumi et al, 2021), including sentences, paragraphs, and even entire papers, while upholding grammatical consistency and human language rules . GPT models' main feature is their capacity to be pre-trained on substantial volumes of textual data (Markel et al, 2023) and then adjusted for certain tasks that come after, including text categorization or question answering (Floridi & Chiriatti, 2020).…”

“…ChatGPT has the capacity to impact a substantial segment of the world's populace (Haleem et l., 2022). The field of Natural Language Processing (NLP) (Peng et al, 2023;Adewumi et al, 2021), a subfield of Artificial Intelligence (AI), is where ChatGPT first emerged (Remountakis et al, 2023). NLP is devoted to helping machines understand and produce human language (Remountakis et al, 2023).…”

T5 for Hate Speech, Augmented Data, and Ensemble

“…The second technique involves the use of the dialogue (conversation) model checkpoint by Adewumi et al [8], which was finetuned on the Multi-Domain Wizard-of-Oz (MultiWOZ) dataset by Eric et al [46]. It is an autoregressive model based on the pretrained DialoGPT-medium model by Zhang et al [7].…”

“…Additional hyperparameters include maximum decoding length, set to 200 tokens; temperature, set to 0.8; and maximum ngram repeat limit, set to 3. These hyperparameters are based on previous work, as they have been shown to perform well [8].…”

“…Automatic hate speech detection is, therefore, crucial and has been gaining increasing importance because of the rising influence of social media. It will facilitate the elimination/prevention of undesirable characteristics in data and, by extension, AI technologies, such as conversational systems [7,8]. HS examples that may incite others to violence in the Offensive Language Identification Dataset (OLID) [9] are given in Table 1.…”