Lip-Interact

Sun, Ke; Yu, Chun; Shi, Weinan; Liu, Lan; Shi, Yuanchun

doi:10.1145/3242587.3242599

Cited by 99 publications

(28 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dove et al [209] surveyed to understand how design innovation is practiced in the ML domain in terms of user experience. [63], [70], [165], [174], [211] [124], [125], [128], [175], [217] [69], [112], [113], [118], [130], [131], [133], [135], [137], [140], [144], [152], [153], [155] [49], [57], [67], [98], [110], [114], [180], [192], [193], [203], [214], [215] [48], [59], [62], [75], [79], [111], [120], [156], [159], [185], [195], [197], [207], [218] [65], [83],…”

Section: Survey Resultsmentioning

confidence: 99%

“…Lip-Interact [135] is a tool to provide silent voice commands via lip movement. Code-Mend [136] is a tool to support search and integration of code for programmers.…”

Section: Applicationsmentioning

confidence: 99%

“…Given that the majority in those sectors are not AI experts developing AI systems for them requires us to investigate the human aspect of such systems. Our analysis shows that application domains have specifically targeted gaming [63,70,165,174,211], interactive technologies [69,112,113,118,130,131,[133][134][135]137,140,144,152,153,155,212,213], medicine [49,57,67,110,114,180,192,193,203,214,215], psychiatry [98], music [65,83,85,102,153], sports [126], dating [60], video production [84], assistive technologies [2,80,86,88,89,…”

Section: Application Domainsmentioning

confidence: 99%

See 2 more Smart Citations

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

Kaluarachchi

Reis

Nanayakkara

2021

Sensors

View full text Add to dashboard Cite

After Deep Learning (DL) regained popularity recently, the Artificial Intelligence (AI) or Machine Learning (ML) field is undergoing rapid growth concerning research and real-world application development. Deep Learning has generated complexities in algorithms, and researchers and users have raised concerns regarding the usability and adoptability of Deep Learning systems. These concerns, coupled with the increasing human-AI interactions, have created the emerging field that is Human-Centered Machine Learning (HCML). We present this review paper as an overview and analysis of existing work in HCML related to DL. Firstly, we collaborated with field domain experts to develop a working definition for HCML. Secondly, through a systematic literature review, we analyze and classify 162 publications that fall within HCML. Our classification is based on aspects including contribution type, application area, and focused human categories. Finally, we analyze the topology of the HCML landscape by identifying research gaps, highlighting conflicting interpretations, addressing current challenges, and presenting future HCML research opportunities.

show abstract

Section: Survey Resultsmentioning

confidence: 99%

“…Lip-Interact [135] is a tool to provide silent voice commands via lip movement. Code-Mend [136] is a tool to support search and integration of code for programmers.…”

Section: Applicationsmentioning

confidence: 99%

Section: Application Domainsmentioning

confidence: 99%

See 1 more Smart Citation

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

Kaluarachchi

Reis

Nanayakkara

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Some works have proposed the use of camera-based techniques for developing SSIs, e.g. LipNet [1], Lip-Interact [27], so that no special sensors are required. However, an additional camera needs to be installed towards the user's mouth to capture the user's lip movements.…”

Section: Related Workmentioning

confidence: 99%

CELIP: Ultrasonic-based Lip Reading with Channel Estimation Approach for Virtual Reality Systems

Zhang

Chen

Wang

et al. 2021

Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of The

View full text Add to dashboard Cite

We developed an ultrasonic-based silent speech interface for Virtual Reality (VR). As more and more customized devices are proposed to enhance the immersion and experience of VR, our system can be used to improve the capability of interactions between users and the systems, while retaining the possibilities of using various customized devices and avoiding some limitations of traditional speech recognition. By employing the channel estimation techniques with ultrasonic waves, we can derive movement characteristics of users' lips, which can be used to fine-tune existing speech recognition models and augmented by vast open-sourced speech datasets. Moreover, we use the speech interface to guide the initialization of customized models for new users, so that they can easily have the access to our system. A two-stage experiment has been conducted and the results show that our system can achieve 90.8% commandlevel accuracy and 1.3% word-error-rate in sentence-level accuracy. CCS CONCEPTS• Human-centered computing → Interaction techniques.

show abstract

“…AVSR system can be applied to a vastly wider range of applicating scenarios, such as command recognition in land vehicles, 4 text translating of mobile phones, 5,6 lipreading for people who are hearing impaired, 7,8 speech recognition of individual speaker from lots of people who are talking at once, 9 and so on. In practice, the difficulties in building an efficient AVSR system mainly concentrate on the following three aspects: How to build public databases for general purpose. How to extract the representative visual features for a specific use. How to dynamically fuse audio features and visual features for a better speech recognition. …”

Section: Introductionmentioning

confidence: 99%

Audiovisual speech recognition: A review and forecast

Xia

Chen

et al. 2020

International Journal of Advanced Robotic Systems

View full text Add to dashboard Cite

Audiovisual speech recognition is a favorable solution to multimodality human–computer interaction. For a long time, it has been very difficult to develop machines capable of generating or understanding even fragments of natural languages; the fused sight, smelling, touching, and so on provide machines with possible mediums to perceive and understand. This article presents a detailed review of recent advances in audiovisual speech recognition area. After explicitly representing audiovisual speech recognition development phase divided by timeline, we focus on typical audiovisual speech database descriptions in terms of single view and multi-view, since the public databases for general purpose should be the first concern for audiovisual speech recognition tasks. For the following challenges that are inseparably related to the feature extraction and dynamic audiovisual fusion, the principal usefulness of deep learning-based tools, such as deep fully convolutional neural network, bidirectional long short-term memory network, 3D convolutional neural network, and so on, lies in the fact that they are relatively simple solutions of such problems. As the principle analyses and comparisons related to computational load, accuracy, and applicability of well-developed audiovisual speech recognition frameworks have been conducted, we further illuminate our insights into the future audiovisual speech recognition architecture design. We argue that end-to-end audiovisual speech recognition model and deep learning-based feature extractors will guide multimodality human–computer interaction directly to a solution.

show abstract

Lip-Interact

Cited by 99 publications

References 34 publications

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

CELIP: Ultrasonic-based Lip Reading with Channel Estimation Approach for Virtual Reality Systems

Audiovisual speech recognition: A review and forecast

Contact Info

Product

Resources

About