Multi-Platform Intelligent System for Multimodal Human-Computer Interaction

Jarosz, Mateusz; Nawrocki, Piotr; Śnieżyński, Bartłomiej; Indurkhya, Bipin

doi:10.31577/cai_2021_1_83

Cited by 8 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human-robot interaction refers to the ability to communicate and interact between humans and machines, and the prerequisite for achieving this activity is that the robot is able to accurately read the human language, behavior, action intentions, etc. [1][2][3]. Currently, the carriers of human-machine interaction are mostly intelligent robots.…”

Section: Introductionmentioning

confidence: 99%

“…(1) Discrepancies in scope (2) Discrepancies in the description of the research reported (3) Discrepancies between the availability of data and the research described (4) Inappropriate citations (5) Incoherent, meaningless and/or irrelevant content included in the article (6) Manipulated or compromised peer review Te presence of these indicators undermines our confdence in the integrity of the article's content and we cannot, therefore, vouch for its reliability. Please note that this notice is intended solely to alert readers that the content of this article is unreliable.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A Multimodal Information Fusion Model for Robot Action Recognition with Time Series

Zhang¹,

Li²,

Qian³

2022

Journal of Electrical and Computer Engineering

View full text Add to dashboard Cite

The current robotics field, led by a new generation of information technology, is moving into a new stage of human-machine collaborative operation. Unlike traditional robots that need to use isolation rails to maintain a certain safety distance from people, the new generation of human-machine collaboration systems can work side by side with humans without spatial obstruction, giving full play to the expertise of people and machines through an intelligent assignment of operational tasks and improving work patterns to achieve increased efficiency. The robot’s efficient and accurate recognition of human movements has become a key factor in measuring robot performance. Usually, the data for action recognition is video data, and video data is time-series data. Time series describe the response results of a certain system at different times. Therefore, the study of time series can be used to recognize the structural characteristics of the system and reveal its operation law. As a result, this paper proposes a time series-based action recognition model with multimodal information fusion and applies it to a robot to realize friendly human-robot interaction. Multifeatures can characterize data information comprehensively, and in this study, the spatial flow and motion flow features of the dataset are extracted separately, and each feature is input into a bidirectional long and short-term memory network (BiLSTM). A confidence fusion method was used to obtain the final action recognition results. Experiment results on the publicly available datasets NTU-RGB + D and MSR Action 3D show that the method proposed in this paper can improve action recognition accuracy.

show abstract

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

A Multimodal Information Fusion Model for Robot Action Recognition with Time Series

Zhang¹,

Li²,

Qian³

2022

Journal of Electrical and Computer Engineering

View full text Add to dashboard Cite

show abstract

“…With the progress of science and technology, many pioneers of technology are trying to combine voice, vision, text, and other information, that is, multimodal information, to promote the upgrade of Human-Computer Interaction (HCI) technology. Multimodal interaction has also become a hot topic in academia and industry [1]. Multimodal technology will not be limited to speech and visual recognition but will gradually change the whole world in this revolution.…”

Section: Introductionmentioning

confidence: 99%

Deep Learning for Intelligent Human–Computer Interaction

Poiesi

Dong

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.

show abstract