Multimodal fusion for robotics

Liu, Huaping; Deng, Chenwei; Fernández-Caballero, Antonio; Sun, Fuchun

doi:10.1177/1729881418782832

Cited by 5 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A potential way to equip the robots with a high level of interaction capabilities is to explore human's underlying sensorimotor principles and integrate multimodal information into the robotic control policies [11][12][13]. Neurological research has shown that humans can adapt limb impedance subconsciously to deal with different situations when performing tasks thanks to the central nervous system (CNS).…”

Section: Introductionmentioning

confidence: 99%

Simultaneously Encoding Movement and sEMG-Based Stiffness for Robotic Skill Learning

Zeng

Yang

Cheng

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 99%

Simultaneously Encoding Movement and sEMG-Based Stiffness for Robotic Skill Learning

Zeng

Yang

Cheng

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

show abstract

“…The function is known as the "probability mass function" of the sensor Si, indicated by mi. So, with respect to sensor Si's notice, the probability which "the detected person is user A" is specified by a "confident interval," as illustrated in equation (8).…”

Section: Dempster-shafer Theory (Dst)mentioning

confidence: 99%

“…Data fusion is the joint analysis of multiple inter-related datasets that provide complementary views of the same phenomenon [6]. Data fusion systems are now widely used in various areas such as sensor networks [7], robotics [8], video and image processing [9], and intelligent system design [10], etc. The statistics of recent digital information around the world estimates 80%-90% of data generated by digitized services via industry is unstructured [10].…”

Section: Introductionmentioning

confidence: 99%

An Adaptive and Late Multimodal Fusion Framework in Contextual Representation based Evidential Deep Learning Dempster-Shafer Theory

Eldin,

Hassanein,

Hassanien

2023

Preprint

View full text Add to dashboard Cite

There is a growing interest in multidisciplinary research in multimodal synthesis technology to stimulate diversity of modal interpretation in different application contexts. The current literature review focuses on modality-based systems in a specific known context and leaves a gap in fusing multiple modality data types in different contexts. Therefore, there seems to be a real requirement for an analytical review of recent developments in the field of data fusion. The real requirement for modality diversity across multiple contextual representation fields is due to the conflicting nature of data in multi-target sensors, which introduces other obstacles including ambiguous, uncertain data, imbalance and redundancy in multi object classification. Additionally, there is lack of frameworks that can analyze offline stream data to identify hidden relationships between different modal data types and different modal counts. Additionally, the lack of a multimodal fusion model capable of determining the extraction conditions of the extracted fusion data has led to low accuracy rates in classifying objects across modalities and systems. This paper proposes a new adaptive and late multimodal fusion framework to interpret multiple modalities and contextual representations using evidence-enhanced deep learning-based Dempster-Shafer theory. The proposed multimodal fusion framework is a MultiFusion learning model solution to solve the Modality-and context-based fusion to improve remote management, intelligent systems, and decision making. The proposed multimodal fusion framework can address the contradictory nature of data uncertainty, diversity of methods, factors, conditions, and relationships for multimodal explanation in multi-context systems to improve decision making and control in diverse contextual representations. Furthermore, this research provides a comparative analysis of the current fusion and prior multimodal data fusion models, explaining the differences of the construction analysis, mathematical analysis of fusion models, pros, and cons of them. In addition, this research presents a comparative analysis between the proposed framework with previous published fusion frameworks, exploring their concepts, advantages and problems, drivers, and current techniques. The experimental accuracy results in multimodalities experiments and multi-context using the proposed multimodal fusion framework is 98.45%. Additionally, some future research directions are discussed.

show abstract

“…To fully verify the validity of the proposed HDDPDI video representation as well as to compare the influences of different CNN layers on action recognition, we design three classification schemes in the recognition framework where different CNN layers are used. Since three projected views can offer complementary characteristics for human actions, multimodal information fusion [32][33][34] is applied in each classification scheme and results of three views are combined for action recognition. The proposed action recognition framework is described in Figure 1.…”

Section: Introductionmentioning

confidence: 99%

Hierarchical dynamic depth projected difference images–based action recognition in videos with convolutional neural networks

2019

International Journal of Advanced Robotic Systems

View full text Add to dashboard Cite

Temporal information plays a significant role in video-based human action recognition. How to effectively extract the spatial-temporal characteristics of actions in videos has always been a challenging problem. Most existing methods acquire spatial and temporal cues in videos individually. In this article, we propose a new effective representation for depth video sequences, called hierarchical dynamic depth projected difference images that can aggregate the action spatial and temporal information simultaneously at different temporal scales. We firstly project depth video sequences onto three orthogonal Cartesian views to capture the 3D shape and motion information of human actions. Hierarchical dynamic depth projected difference images are constructed with the rank pooling in each projected view to hierarchically encode the spatial-temporal motion dynamics in depth videos. Convolutional neural networks can automatically learn discriminative features from images and have been extended to video classification because of their superior performance. To verify the effectiveness of hierarchical dynamic depth projected difference images representation, we construct a hierarchical dynamic depth projected difference images-based action recognition framework where hierarchical dynamic depth projected difference images in three views are fed into three identical pretrained convolutional neural networks independently for finely retuning. We design three classification schemes in the framework and different schemes utilize different convolutional neural network layers to compare their effects on action recognition. Three views are combined to describe the actions more comprehensively in each classification scheme. The proposed framework is evaluated on three challenging public human action data sets. Experiments indicate that our method has better performance and can provide discriminative spatial-temporal information for human action recognition in depth videos.

show abstract

Multimodal fusion for robotics

Cited by 5 publications

References 24 publications

Simultaneously Encoding Movement and sEMG-Based Stiffness for Robotic Skill Learning

Simultaneously Encoding Movement and sEMG-Based Stiffness for Robotic Skill Learning

An Adaptive and Late Multimodal Fusion Framework in Contextual Representation based Evidential Deep Learning Dempster-Shafer Theory

Hierarchical dynamic depth projected difference images–based action recognition in videos with convolutional neural networks

Contact Info

Product

Resources

About