Survey of Fusion Techniques for Design of Efficient Multimodal Systems

Chhabria, S. A.; Dharaskar, Rajiv V.; Thakare, V. M.

doi:10.1109/icmira.2013.103

Cited by 8 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[20], [21], and [22] are three recent survey works of multimodal fusion, and they classify approaches including different modalities. Articles [6], [23], [24], and [25] present an overview of the methods of fusion with challenges and prospects. Instead of focusing on fusion processes, there are surveys related to the use of fusion in several domains.…”

Section: Related Studiesmentioning

confidence: 99%

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

2023

View full text Add to dashboard Cite

Multimodal machine learning (MML) is a tempting multidisciplinary research area where heterogeneous data from multiple modalities and machine learning (ML) are combined to solve critical problems. Usually, research works use data from a single modality, such as images, audio, text, and signals. However, real-world issues have become critical now, and handling them using multiple modalities of data instead of a single modality can significantly impact finding solutions. ML algorithms play an essential role by tuning parameters in developing MML models. This paper reviews recent advancements in the challenges of MML, namely: representation, translation, alignment, fusion and co-learning, and presents the gaps and challenges. A systematic literature review (SLR) applied to define the progress and trends on those challenges in the MML domain. In total, 1032 articles were examined in this review to extract features like source, domain, application, modality, etc. This research article will help researchers understand the constant state of MML and navigate the selection of future research directions.

show abstract

Section: Related Studiesmentioning

confidence: 99%

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

2023

View full text Add to dashboard Cite

show abstract

“…Multimodal fusion involves fusing multiple senses and utilizing more than one input channel, i.e., gestures, speech, vision, touch, etc., in one system to interact with the machine. Chhabria et al [12] demonstrated that the use of multimodal methods in virtual reality scenarios enhances the naturalness and efficiency of interactions compared to single-modality techniques.…”

Section: A Multimodal Fusion Approach To Intent Understandingmentioning

confidence: 99%

MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments

et al. 2023

View full text Add to dashboard Cite

The current virtual system for secondary school experiments poses several issues, such as limited methods of operation for students and an inability of the system to comprehend the users’ operational intentions, resulting in a greater operational burden for students and hindering the goal of the experimental practice. However, many traditional multimodal fusion algorithms rely solely on individual modalities for the analysis of users’ experimental intentions, failing to fully utilize the intention information for each modality. To rectify these issues, we present a new multimodal fusion algorithm, MFIRA, which intersects and blends intention probabilities between channels by executing parallel processing of multimodal information at the intention layer. Additionally, we developed an augmented reality (AR) virtual experiment platform based on the Hololens 2, which enables students to conduct experiments using speech, gestures, and vision. Employing the MFIRA algorithm, the system captures users’ experimental intent and navigates or rectifies errors to guide students through their experiments. The experimental results indicate that the MFIRA algorithm boasts a 97.3% accuracy rate in terms of interpreting users’ experimental intent. Compared to existing experimental platforms, this system is considerably more interactive and immersive for students and is highly applicable in secondary school experimental chemistry classrooms.

show abstract

“…to communicate with a machine in a system is referred to as multimodal interaction. As a result, adopting multimodal input interaction in virtual-reality settings might improve the naturalness and efficiency of interaction compared to using unimodal [ 8 ].…”

Section: Related Workmentioning

confidence: 99%

MFA: A Smart Glove with Multimodal Intent Sensing Capability

Wang

Feng

Tian

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

At present, virtual-reality fusion smart experiments mostly employ visual perception devices to collect user behavior data, but this method faces the obstacles of distance, angle, occlusion, light, and a variety of other factors of indoor interactive input devices. Moreover, the essence of the traditional multimodal fusion algorithm (TMFA) is to analyze the user’s experimental intent serially using single-mode information, which cannot fully utilize the intent information of each mode. Therefore, this paper designs a multimodal fusion algorithm (hereinafter referred to as MFA, Algorithm 4) which focuses on the parallel fusion of user’s experimental intent. The essence of the MFA is the fusion of multimodal intent probability. At the same time, this paper designs a smart glove based on the virtual-reality fusion experiments, which can integrate multichannel data such as voice, visual, and sensor. This smart glove can not only capture user’s experimental intent but also navigate, guide, or warn user’s operation behaviors, and it has stronger perception capabilities compared to any other data glove or smart experimental device. The experimental results demonstrate that the smart glove presented in this paper can be widely employed in the chemical experiment teaching based on virtual-reality fusion.

show abstract

Survey of Fusion Techniques for Design of Efficient Multimodal Systems

Cited by 8 publications

References 0 publications

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

MFIRA: Multimodal Fusion Intent Recognition Algorithm for AR Chemistry Experiments

MFA: A Smart Glove with Multimodal Intent Sensing Capability

Contact Info

Product

Resources

About