Evaluating the language resources of chatbots for their potential in English as a second language

Abstract

This paper investigates the linguistic worth of current ‘chatbot’ programs – software programs which attempt to hold a conversation, or interact, in English – as a precursor to their potential as an ESL (English as a second language) learning resource. After some initial background to the development of chatbots, and a discussion of the Loebner Prize Contest for the most ‘human’ chatbot (the ‘Turing Test’), the paper describes an in-depth study evaluating the linguistic accuracy of a number of chatbots available online. Since the ultimate purpose of the current study concerns chatbots' potential with ESL learners, the analysis of language embraces not only an examination of features of language from a native-speaker's perspective (the focus of the Turing Test), but also aspects of language from a second-language-user's perspective. Analyses indicate that while the winner of the 2005 Loebner Prize is the most able chatbot linguistically, it may not necessarily be the chatbot most suited to ESL learners. The paper concludes that while substantial progress has been made in terms of chatbots' language-handling, a robust ESL ‘conversation practice machine’ (Atwell, 1999) is still some way off being a reality.

Conversation System with State Information

“…A perceived objective of a RDR-based conversation system is to break free of the current truism that conversation systems that work within a particular domain (or topic) are more effective. 9) Conversation systems covering a particular topic are, however, likely to be highly desirable in the commercial settings for which such rapid conversation system development seems tailored, and the system can still demonstrate its main strengths in this area by allowing very fine-grained control of subtopics within a domain. In this context it would be considered acceptable to tailor systems to a certain extent towards the objectives they intend to achieve.…”

“…This generally reveals that they merely respond to the statements provided by the human participant. 9) Systems that do implement topic awareness do not treat this as a primary objective or impose heavy structure over the process. 8) 3.…”

“…It can be seen that the only operation that will be performed on user-entered text is basic pattern-matching, and that responses are not generated dynamically. The result of this is that the grammatically garbled responses that are a frequently observed result of stringent testing of a conversation system 9) will not occur. Implementing sufficient appearance of "language diversity" in such a system could be of future concern, but this is not a primary objective of this study.…”

Tareas 2.0 para el aprendizaje y evaluación de segundas lenguas en entornos virtuales de aprendizaje.

“…Asimismo, la recogida de datos cualitativos se ha realizado mediante el análisis de narraciones y ensayos críticos por medio de la herramienta virtual de foros. Esta actividad forma parte de la evaluación continua de la asignatura del Máster y se encaminó a recoger las apreciaciones del alumnado con respecto al uso de herramientas 2.0 en la enseñanza-aprendizaje de segundas lenguas y los procesos asociados a su evaluación; lo que muestra un gran valor para los propios actores (Chase, 1995). La participación en foros de discusión en línea como tarea académica dentro del desarrollo de la asignatura del Máster; así como la respuesta al cuestionario produce una narración extensa de tipo descriptivo y vivencial, en la que afloran los hechos y emociones que han sido más representativos y cruciales para los implicados.…”

“…Finalmente, se ha utilizado el programa EXCEL para concretar el recuento de frecuencias de las subcategorías, de forma individual (para cada participante y código de análisis) y globalmente (atendiendo a las respectivas subcategorías y al conjunto de los participantes). La codificación de los análisis permitió profundizar en el contenido de las narrativas, considerando tanto sus aspectos objetivables como los de carácter subjetivo y emocional (Clandinin & Connelly, 1994 …”