Multimodal HALEF: An Open-Source Modular Web-Based Multimodal Dialog Framework

Yu, Zhou; Ramanarayanan, Vikram; Mundkowsky, Robert; Lange, Patrick; Ivanov, Alexei V.; Black, Alan W.; Suendermann-Oeft, David

doi:10.1007/978-981-10-2585-3_18

Cited by 7 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HALEF is an open-source, modular, cloud-based dialog system that is compatible with multiple W3C and open industry standards. The HALEF architecture and components have been described in detail in prior publications [13,14,15].…”

Section: The Halef Dialog Ecosystemmentioning

confidence: 99%

Human and Automated Scoring of Fluency, Pronunciation and Intonation During Human–Machine Spoken Dialog Interactions

et al. 2017

Self Cite

View full text Add to dashboard Cite

We present a spoken dialog-based framework for the computerassisted language learning (CALL) of conversational English. In particular, we leveraged the open-source HALEF dialog framework to develop a job interview conversational application. We then used crowdsourcing to collect multiple interactions with the system from non-native English speakers. We analyzed human-rated scores of the recorded dialog data on three different scoring dimensions critical to the delivery of conversational English-fluency, pronunciation and intonation/stress-and further examined the efficacy of automatically-extracted, hand-curated speech features in predicting each of these subscores. Machine learning experiments showed that trained scoring models generally perform at par with the human inter-rater agreement baseline in predicting human-rated scores of conversational proficiency.

show abstract

Section: The Halef Dialog Ecosystemmentioning

confidence: 99%

Human and Automated Scoring of Fluency, Pronunciation and Intonation During Human–Machine Spoken Dialog Interactions

et al. 2017

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Audiovisual Dialog Dataset Generationmentioning

confidence: 99%

Rushing to Judgement: How do Laypeople Rate Caller Engagement in Thin-Slice Videos of Human–Machine Dialog?

2017

Self Cite

View full text Add to dashboard Cite

We analyze the efficacy of a small crowd of naïve human raters in rating engagement during human-machine dialog interactions. Each rater viewed multiple 10 second, thin-slice videos of non-native English speakers interacting with a computerassisted language learning (CALL) system and rated how engaged and disengaged those callers were while interacting with the automated agent. We observe how the crowd's ratings compared to callers' self ratings of engagement, and further study how the distribution of these rating assignments vary as a function of whether the automated system or the caller was speaking. Finally, we discuss the potential applications and pitfalls of such a crowdsourced paradigm in designing, developing and analyzing engagement-aware dialog systems.

show abstract

“…Kuansan et al proposed a multimodal dialogue framework that used the inputted multimodal interaction data to infer dialogue intentions of users [14]. Zhou et al proposed a multimodal dialogue framework that included video conference capabilities [15]. Although there have been several studies on multimodal dialogue systems, their incorporation into language learning has not been widely reported.…”

Section: Related Workmentioning

confidence: 99%

Improving Language Learning Performance Using Multimodal Dialogue Systems

Wu,

Ding,

Zhang

et al. 2023

Atlantis Highlights in Computer Sciences

View full text Add to dashboard Cite

Learning a language poses a significant challenge for learners, who must comprehend the intricacies of language and analyze the relationships among its components. While computer technologies have been developed to assist language learning, they fail to mirror the human cognitive process. This paper examines the application of a multimodal dialogue system to enhance language learning outcomes. The system boasts several advantages. Firstly, smart devices can collect multimodal data in learning environments to monitor the learner's status in realtime, thus enhancing the accuracy of intention recognition. Secondly, the system can interact with learners naturally by analyzing their multimodal data, resulting in improved language skills. Finally, application scenarios are designed based on the defined multimodal dialogue system, which effectively demonstrates the system's ability to enhance language learning performance.

show abstract

Multimodal HALEF: An Open-Source Modular Web-Based Multimodal Dialog Framework

Cited by 7 publications

References 15 publications

Human and Automated Scoring of Fluency, Pronunciation and Intonation During Human–Machine Spoken Dialog Interactions

Human and Automated Scoring of Fluency, Pronunciation and Intonation During Human–Machine Spoken Dialog Interactions

Rushing to Judgement: How do Laypeople Rate Caller Engagement in Thin-Slice Videos of Human–Machine Dialog?

Improving Language Learning Performance Using Multimodal Dialogue Systems

Contact Info

Product

Resources

About