MiPad: a multimodal interaction prototype

Huang, Xuedong; Acero, Alejandro; Chelba, Ciprian; Deng, Li; Droppo, Jasha; Duchene, Doug; Goodman, Joshua; Hon, Hsiao‐Wuen; Jacoby, Derek; Jiang, Li; Loynd, Ricky; Mahajan, Milind; Mau, Peter; Meredith, Scott; Mughal, Salman; Neto, Salvado; Plumpe, Mike; Steury, Keith; Venolia, Gina; Wang, K.; Wang, Y.

doi:10.1109/icassp.2001.940754

Cited by 30 publications

(25 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, whenever hands are occupied (e.g., while driving), or where hand-based interfaces are bulky (using personal digital assistances (PDAs) or cell phones), ASR will undeniably succeed. Indeed, ASR is increasingly used on hand-held devices [14] -some PDA-based ASR systems are starting to appear commercially such as the IBM personal speech assistant [4] and the Microsoft MiPad [8] (others are listed in [14]). A number of wireless communication companies also launched their products integrated with ASR systems, like Motorala's voiceXML and Nokia 9000 series.…”

Section: Introductionmentioning

confidence: 99%

Algorithms for data-driven ASR parameter quantization

Filali¹,

Li²,

Bilmes³

2006

Computer Speech & Language

View full text Add to dashboard Cite

Section: Introductionmentioning

confidence: 99%

Algorithms for data-driven ASR parameter quantization

Filali¹,

Li²,

Bilmes³

2006

Computer Speech & Language

View full text Add to dashboard Cite

“…Before deep learning methods were adopted, there had already been numerous efforts in multimodal and multitask learning. For example, a prototype called MiPad for multimodal interactions involving capturing, leaning, coordinating, and rendering a mix of speech, touch, and visual information was developed and reported in [113,164]. In [165,166], mixed sources of information from multiple-sensory microphones with separate bone-conductive and air-born paths were exploited to de-noise speech.…”

Section: B) a Selected Review On Deep Learning For Multimodal Processingmentioning

confidence: 99%

“…A wide range of noise-robust techniques developed over past 30 years can be analyzed and categorized using five different criteria: (1) feature-domain versus model-domain processing, (2) the use of prior knowledge about the acoustic environment distortion, (3) the use of explicit environment-distortion models, (4) deterministic versus uncertainty processing, and (5) the use of acoustic models trained jointly with the same feature enhancement or model adaptation process used in the testing stage. See a comprehensive review in [109,110] and additional review literature or original work in [111][112][113][114].…”

Section: D) a New Level Of Noise Robustnessmentioning

confidence: 99%

Deep learning: from speech recognition to language and multimodal processing

Deng

2016

SIP

View full text Add to dashboard Cite

While artificial neural networks have been in existence for over half a century, it was not until year 2010 that they had made a significant impact on speech recognition with a deep form of such networks. This invited paper, based on my keynote talk given at Interspeech conference in Singapore in September 2014, will I . I N T R O D U C T I O NThe main theme of this paper is to reflect on the recent history of how deep learning has profoundly revolutionized the field of automatic speech recognition (ASR) and to elaborate on what kind of lessons we can learn to not only further advance ASR technology but also to impact the related, arguably more important, applications in language and multimodal processing. Language processing concerns "downstream" analysis and distillation of information from the ASR systems' outputs. Semantic analysis of language and multimodal processing involving speech, text, and image, both experiencing rapid advances based on deep learning over the past few years, holds the potential to solve some difficult and remaining ASR problems and present new challenges for the deep learning technology.A message to be conveyed in this paper is the importance of broadening deep learning from deep neural networks (DNNs) to include deep generative models as well. In fact, a brief historical review conducted in Section II will touch Microsoft Research, One Microsoft Way, Redmond, WA 98052, USA Corresponding author: Li Deng Email: deng@microsoft.com on how the development of deep (and dynamic) generative models of speech played a role in the inroads of DNNs into modern ASR. Since 2011, the DNN has taken over the dominating (shallow) generative model of speech, the Gaussian Mixture Model (GMM), as the output distribution in the Hidden Markov Model (HMM). This purely discriminative DNN has been well-known to the ASR community, which can be considered as a shallow network unfolding in space. When the unfolding occurs in time, we have the recurrent neural network (RNN). On the other hand, deep generative models have distinct advantages over discriminative DNNs, including the strengths of model interpretability, of embedding domain knowledge and causal relationships, and of modeling uncertainty. Deep generative and discriminative models represent two apparently opposing approaches yet with highly complementary strengths and weaknesses. The further success of deep learning is likely to lie in how to seamlessly integrate the two approaches in a practically effective and theoretically appealing fashion, and to achieve the best of both worlds.The remainder of this paper is organized as follows. In Section II, some brief history is provided on how deep learning made inroad into speech recognition, and a number of enabling factors are discussed. Outstanding achievements of deep learning both in academic world and in industry to 1 https://www.cambridge.org/core/terms. https://doi

show abstract

“…Among several use-cases such as mobile interaction, interactive maps, graphic design applications and systems for augmented dual-task environments, mobile systems are selected for detailed discussion, underpinning of pattern candidates and pattern identification. Examples for multimodal mobile interaction are personal assistants for e-mail and web access such as MiPad [22], Personal Speech Assistant [13], tourist guides and city information systems such as SmartKom mobile [26], MATCH [21,24], MUST [3] or COMPASS [4].…”

Section: Identification Of Multimodal User Interface Patterns Based Omentioning

confidence: 99%

“…Mobile Systems such as Microsoft's MiPad [22] and IBM's Personal Speech Assistant [13] are good examples.…”

Section: Known Usesmentioning

confidence: 99%

Steps in Identifying Interaction Design Patterns for Multimodal Systems

Ratzka

Engineering Interactive Systems 2008

View full text Add to dashboard Cite

Abstract. The context of this work is usability engineering for multimodal interaction. In contrast to other work that concentrates on prototyping toolkits or abstract guidelines, this research focuses on user interface patterns for multimodal interaction. Designing multimodal applications requires several skills ranging from design and implementation. Thus, different kinds of patterns (from architecture patterns to user interface patterns) can be applied to this field. This work focuses on user-task near user interface patterns. At first, a traditional approach of modality selection based on task-and context-based rules is presented. Next, a twofold process of pattern mining is presented. In the first phase, pattern candidates are derived top-down from proven knowledge about how multimodality enhances usability. In the second phase, literature is mined for real solutions to underpin these pattern candidates and find new ones. Along with this, relationships between patterns are depicted.

show abstract

MiPad: a multimodal interaction prototype

Cited by 30 publications

References 5 publications

Algorithms for data-driven ASR parameter quantization

Algorithms for data-driven ASR parameter quantization

Deep learning: from speech recognition to language and multimodal processing

Steps in Identifying Interaction Design Patterns for Multimodal Systems

Contact Info

Product

Resources

About