Combining User Modeling and Machine Learning to Predict Users’ Multimodal Integration Patterns

Xiao, Huang; Oviatt, Sharon; Lunsford, Rebecca

doi:10.1007/11965152_5

Cited by 20 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There has been significant work attempting to model users of technology through ML methods (see Webb, Pazzani, and Billsus [2001] for a review), but with little to no attention paid to how such models can inform system design. For example, Huang, Oviatt, and Lunsford (2006) developed a user model based on ML to better predict users' multimodal integration patterns via speech and a pen to develop a system that dynamically responded to user behaviors. However, even though they were successful in developing a predictive model, this model was never actually applied to the design of a system.…”

Section: Design Applications Of MLmentioning

confidence: 99%

Identifying Critical Contextual Design Cues Through a Machine Learning Approach

Cummings

Stimpson²

2019

AI Magazine

View full text Add to dashboard Cite

Given the rise of autonomous systems in transportation, medical, and manufacturing industries, there is an increasing need to understand how such systems should be designed to promote effective interactions between one or more humans working in and around these systems. Practitioners often have difficulties in conducting costly and time-consuming human-in-the-loop studies, so an analytical strategy that helps them determine whether their designs are capturing their planned intent is needed. A traditional top-down, hypothesis-driven experiment that examined whether external displays mounted on autonomous cars could effectively communicate with pedestrians led to the conclusion that the displays had no effect on safety. However, by first taking a bottom-up, data-driven machine learning approach, those segments of the population that were most affected by the external displays were identified. Then, a hypothesis-driven, within-subjects analysis of variance revealed that an external display mounted on an autonomous car that provided the vehicle’s speed as opposed to commanding a go/no-go decision provided an additional 4 feet of safety for early adopters. One caveat to this approach is that the selection of a specific algorithm can significantly influence the results and more work is needed to determine the sensitivity of this approach with seemingly similar machine learning classification approaches.

show abstract

Section: Design Applications Of MLmentioning

confidence: 99%

Identifying Critical Contextual Design Cues Through a Machine Learning Approach

Cummings

Stimpson²

2019

AI Magazine

View full text Add to dashboard Cite

show abstract

“…Integration patterns are also investigated in [3,4], were machine learning based approaches to predict a user's integration pattern were presented, based on the data of [7]. It is shown that 15 samples per user are enough to predict a user's integration pattern with an accuracy of 81%, so integration patterns can be detected relatively fast, even in an automated way.…”

Section: Related Workmentioning

confidence: 99%

“…The main measures, that may be relevant, are the input order (touch before speech, or speech before touch) and the time distances between the two modalities. The analyzed 11 participants split up into two groups, namely those, that always started with speech input (7), and those that always started with touch input (4). None of them changed the input order during the experiment.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Interaction History and its use in Error Detection and Recovery

Schüssel

Honold

Schmidt

et al. 2014

Proceedings of the 16th International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Multimodal systems still tend to ignore the individual input behavior of users, and at the same time, suffer from erroneous sensor inputs. Although many researchers have described user behavior in specific settings and tasks, little to nothing is known about the applicability of such information, when it comes to increase the robustness of a system for multimodal inputs. We conducted a gamified experimental study to investigate individual user behavior and error types found in an actually running system. It is shown, that previous ways of describing input behavior by a simple classification scheme (like simultaneous and sequential) are not suited to build up an individual interaction history. Instead, we propose to use temporal distributions of different metrics derived from multimodal event timings. We identify the major errors that can occur in multimodal interactions and finally show how such an interaction history can practically be applied for error detection and recovery. Applying the proposed approach to the experimental data, the initial error rate is reduced from 4.9% to a minimum of 1.2%.

show abstract

“…speech and gestures should ideally be adapted to the user and the context (see [Huang et al 2006] for a machine learning approach to that).…”

Section: Recognizing Speech and Gestures 17mentioning

confidence: 99%

Multimodal gesture recognition

Katsamanis¹,

Pitsikalis²,

Theodorakis³

et al. 2017

The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 1

View full text Add to dashboard Cite

2 Chapter 11 Multimodal Gesture Recognition Figure 11.1 Screenshot from the "Put That There!" demonstration video by the Architecture Machine Group at MIT [Bolt 1980]. ers [Kinect 2016]. Such advancements have led to intensified efforts to integrate multimodal gesture interfaces in real-life applications. Indeed, the field of multimodal gesture recognition has been attracting increasing interest, being driven by novel HCI paradigms on a continuously expanding range of devices equipped with multimodal sensors and ever-increasing computational power, for example smartphones and smart television sets. Nevertheless, the capabilities of modern multimodal gesture systems remain limited. In particular, the set of gestures accounted for in typical setups is mostly constrained to pointing gestures, a number of emblematic ones like an open palm, and gestures corresponding to some sort of interaction with a physical object, e.g., pinching for zooming. At the same time, fusion with speech remains in most cases just an experimental feature. When compared to the abundance and variety of gestures and their interaction with speech in natural human communication, it clearly seems that there is still a long way to go for the corresponding HCI research and development [Kopp 2013]. Multimodal gesture recognition constitutes a wide multidisciplinary field. This chapter makes an effort to provide a comprehensive overview of it, both in theoretical and application terms. More specifically, basic concepts related to gesturing, the multifaceted interplay of gestures and speech, and the importance of gestures in HCI are discussed in Section 11.2. An overview of the current trends in the field of multimodal gesture recognition is provided in Section 11.3, separately focusing on gestures, speech, and multimodal fusion. Further, a stateof-the-art recognition setup developed by the authors is described in detail in Section 11.4, in order to facilitate a better understanding of all practical considerations involved in such a system. In closing, the future of multimodal gesture recognition and related challenges are discussed in Section 11.5. Finally, a set of Focus Questions to aid comprehension of the material is also provided. 11.2 Multimodal Communication and Gestures 3 Glossary Terminology for Understanding Multimodal Gesture Recognition Co-speech gestures are gestures produced while speaking. Their interplay with speech and their role in human interaction are discussed in Section 11.2.2. Gesture recognition models are primarily statistical constructs that can be trained to represent specific gestures based on corresponding data. In this chapter, traditional hidden Markov models (HMMs) with Gaussian mixture model (GMM) observation probabilities are considered, as part of the authors' gesture recognition system detailed in Section 11.4. Further, a number of models based on deep learning approaches are overviewed in Section 11.3.1. A more elaborate discussion of deep learning for multimodal interaction modeling can be found in the second vol...

show abstract

Combining User Modeling and Machine Learning to Predict Users’ Multimodal Integration Patterns

Cited by 20 publications

References 13 publications

Identifying Critical Contextual Design Cues Through a Machine Learning Approach

Identifying Critical Contextual Design Cues Through a Machine Learning Approach

Multimodal Interaction History and its use in Error Detection and Recovery

Multimodal gesture recognition

Contact Info

Product

Resources

About