Accuracy and Standardized Judgment Procedures for Author Identification by Text Mining

Zaitsu, Wataru; Jin, Mingzhe

doi:10.2333/jbhmk.45.39

Cited by 2 publications

(3 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To control text lengths, texts containing approximately 1,000 characters were generated by randomly extracting the sentences from each paper, excluding citations. In case of authorship identification using stylometric analysis, texts containing more characters facilitate more accurate identification of authors; however, the minimum number of characters for valid level was determined as approximately 1,000 characters [ 14 , 15 ].…”

Section: Methodsmentioning

confidence: 99%

“…This study included adverbs as function words. Zaitsu & Jin [ 14 ] reported the validity of these four stylometric features in identifying Japanese authors by analyzing texts containing approximately 1,000 characters: the most effective features were the rate of function words; the next was bigrams of parts-of-speech. That study reported high-classification performance levels: 100% on sensitivity and 95.1% on specificity.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis

Zaitsu,

Jin

2023

PLoS ONE

Self Cite

View full text Add to dashboard Cite

In the first half of 2023, text-generative artificial intelligence (AI), including ChatGPT from OpenAI, has attracted considerable attention worldwide. In this study, first, we compared Japanese stylometric features of texts generated by ChatGPT, equipped with GPT-3.5 and GPT-4, and those written by humans. In this work, we performed multi-dimensional scaling (MDS) to confirm the distributions of 216 texts of three classes (72 academic papers written by 36 single authors, 72 texts generated by GPT-3.5, and 72 texts generated by GPT-4 on the basis of the titles of the aforementioned papers) focusing on the following stylometric features: (1) bigrams of parts-of-speech, (2) bigram of postpositional particle words, (3) positioning of commas, and (4) rate of function words. MDS revealed distinct distributions at each stylometric feature of GPT (3.5 and 4) and human. Although GPT-4 is more powerful than GPT-3.5 because it has more parameters, both GPT (3.5 and 4) distributions are overlapping. These results indicate that although the number of parameters may increase in the future, GPT-generated texts may not be close to that written by humans in terms of stylometric features. Second, we verified the classification performance of random forest (RF) classifier for two classes (GPT and human) focusing on Japanese stylometric features. This study revealed the high performance of RF in each stylometric feature: The RF classifier focusing on the rate of function words achieved 98.1% accuracy. Furthermore the RF classifier focusing on all stylometric features reached 100% in terms of all performance indexes (accuracy, recall, precision, and F1 score). This study concluded that at this stage we human discriminate ChatGPT from human limited to Japanese language.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis

Zaitsu,

Jin

2023

PLoS ONE

Self Cite

View full text Add to dashboard Cite

show abstract

“…Function words. Preceding study of authorship attribution [10] and AI detection task [5] reported the function words as quite distinguishable features: "だ (auxiliary verb)," "また (conjunction)," and "は (postpositional particle). "…”

Section: Plos Onementioning

confidence: 99%

Can we spot fake public comments generated by ChatGPT(-3.5, -4)?: Japanese stylometric analysis expose emulation created by one-shot learning

Zaitsu,

Jin,

Ishihara

et al. 2024

PLoS ONE

Self Cite

View full text Add to dashboard Cite

Public comments are an important opinion for civic when the government establishes rules. However, recent AI can easily generate large quantities of disinformation, including fake public comments. We attempted to distinguish between human public comments and ChatGPT-generated public comments (including ChatGPT emulated that of humans) using Japanese stylometric analysis. Study 1 conducted multidimensional scaling (MDS) to compare 500 texts of five classes: Human public comments, GPT-3.5 and GPT-4 generated public comments only by presenting the titles of human public comments (i.e., zero-shot learning, GPTzero), GPT-3.5 and GPT-4 emulated by presenting sentences of human public comments and instructing to emulate that (i.e., one-shot learning, GPTone). The MDS results showed that the Japanese stylometric features of the public comments were completely different from those of the GPTzero-generated texts. Moreover, GPTone-generated public comments were closer to those of humans than those generated by GPTzero. In Study 2, the performance levels of the random forest (RF) classifier for distinguishing three classes (human, GPTzero, and GPTone texts). RF classifiers showed the best precision for the human public comments of approximately 90%, and the best precision for the fake public comments generated by GPT (GPTzero and GPTone) was 99.5% by focusing on integrated next writing style features: phrase patterns, parts-of-speech (POS) bigram and trigram, and function words. Therefore, the current study concluded that we could discriminate between GPT-generated fake public comments and those written by humans at the present time.

show abstract

Accuracy and Standardized Judgment Procedures for Author Identification by Text Mining

Cited by 2 publications

References 5 publications

Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis

Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis

Can we spot fake public comments generated by ChatGPT(-3.5, -4)?: Japanese stylometric analysis expose emulation created by one-shot learning

Contact Info

Product

Resources

About