Comparative analysis of GPT-4, Gemini, and Ernie as gloss sign language translators in special education

Abstract

While several comparative studies have analyzed the effectiveness of various large language models (LLMs), most of them were technical (i.e., comparing execution time, among others). Additionally, these comparative studies did not discuss special education. Consequently, scant information exists about how effective LLMs are in special education. To address this research gap, this study conducted a comparative study of three LLMs, namely GPT-4o, Gemini, and Ernie, as gloss sign language translators for learners with hearing impairments. Specifically, a mixed method was adopted, where the translated outputs of the three LLMs were compared (quantitatively and qualitatively) to two sign language outputs from a sign language expert. The obtained results highlighted that Gemini outperformed both GPT-4o and Ernie as an accurate gloss sign language translator. Additionally, GPT-4o had a high accurate rate, while Ernie had a very low translation performance. The findings of this study can help to raise awareness about the use of LLMs in special education as well as the best ones to use especially with hearing impairment learners.