Gaze-contingent asr for spontaneous, conversational speech: An evaluation

Cooke, Neil; Russell, Melanie

doi:10.1109/icassp.2008.4518639

Cited by 5 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Eye-tracking can be used to re-assign probabilities of speech recognition hypotheses or to adapt the language model, respectively, by considering human's visual attention leading to significant decrease in word error rate [64]. However, achieved better recognition accuracy with such technique was connected more to the visual field than to the visual focus [65]. Eye-tracking and other non-verbal modalities have been combined to make speech recognition more robust against noise [66].…”

Section: Multimodal Integration Of Different Modalities Related To Human-machine Interactionmentioning

confidence: 99%

Integrating Eye- and Mouse-Tracking with Assistant Based Speech Recognition for Interaction at Controller Working Positions

2021

View full text Add to dashboard Cite

Assistant based speech recognition (ABSR) prototypes for air traffic controllers have demonstrated to reduce controller workload and aircraft flight times as a result. However, two aspects of ABSR could enhance benefits, i.e., (1) the predicted controller commands that speech recognition engines use can be more accurate, and (2) the confirmation process of ABSR recognition output, such as callsigns, command types, and values by the controller, can be less intrusive. Both tasks can be supported by unobtrusive eye- and mouse-tracking when using operators’ gaze and interaction data. First, probabilities for predicted commands should consider controllers’ visual focus on the situation data display. Controllers will more likely give commands to aircraft that they focus on or where there was a mouse interaction on the display. Furthermore, they will more likely give certain command types depending on the characteristics of multiple aircraft being scanned. Second, it can be determined via eye-tracking instead of additional mouse clicks if the displayed ABSR output has been checked by the controller and remains uncorrected for a certain amount of time. Then, the output is assumed to be correct and is usable by other air traffic control systems, e.g., short-term conflict alert. If the ABSR output remains unchecked, an attention guidance functionality triggers different escalation levels to display visual cues. In a one-shot experimental case study with two controllers for the two implemented techniques, (1) command prediction probabilities improved by a factor of four, (2) prediction error rates based on an accuracy metric for three most-probable aircraft decreased by a factor of 25 when combining eye- and mouse-tracking data, and (3) visual confirmation of ABSR output promises to be an alternative for manual confirmation.

show abstract

Section: Multimodal Integration Of Different Modalities Related To Human-machine Interactionmentioning

confidence: 99%

Integrating Eye- and Mouse-Tracking with Assistant Based Speech Recognition for Interaction at Controller Working Positions

2021

View full text Add to dashboard Cite

show abstract

“…Eye gaze has been explored in automated language understanding such as speech recognition [4,14], reference resolution [3,13], and recently for word acquisition [10,22]. Given speech paired with eye gaze information and video images, a translation model was used to acquire words by associating acoustic phone sequences with visual representations of objects and actions [22].…”

Section: Related Workmentioning

confidence: 99%

Integrating domain knowledge with user eye gaze in automated word acquisition for conversational interfaces

Chai

2010

Proceedings of the 2010 Workshop on Eye Gaze in Intelligent Human Machine Interaction

View full text Add to dashboard Cite

Most conversation systems tend to fail when unexpected words are encountered. To overcome this problem, conversational systems must be able to learn new words automatically during human machine conversation. Motivated by psycholinguistic findings on eye gaze and human language processing, we have developed several techniques to incorporate human eye gaze for automatic word acquisition in multimodal conversational systems. This paper presents our new results on integrating eye gaze and domain knowledge in word acquisition in a 3D virtual world application. Our results demonstrate that the incorporation of domain knowledge can potentially improve word acquisition performance.

show abstract

Eye Gaze in Intelligent User Interfaces

Nakano¹,

Conati²,

Bader³

2013

View full text Add to dashboard Cite

Gaze-contingent asr for spontaneous, conversational speech: An evaluation

Cited by 5 publications

References 3 publications

Integrating Eye- and Mouse-Tracking with Assistant Based Speech Recognition for Interaction at Controller Working Positions

Integrating Eye- and Mouse-Tracking with Assistant Based Speech Recognition for Interaction at Controller Working Positions

Integrating domain knowledge with user eye gaze in automated word acquisition for conversational interfaces

Eye Gaze in Intelligent User Interfaces

Contact Info

Product

Resources

About