Designing an Augmented Reality Multimodal Interface for 6DOF Manipulation Techniques

Ismail, Ajune Wanis; Billinghurst, Mark; Sunar, Mohd Shahrizal; Yusof, Cik Suhaimi

doi:10.1007/978-3-030-01054-6_22

Cited by 15 publications

(9 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Ismail et al [ 9 ] merged voice and gestures in the context of virtual-reality interaction, making the user's operation of dealing with virtual items in an AR environment more natural. Kadavasal and Oliver [ 10 ] created a virtual-reality driving system for autistic individuals that included physiological signals, brain signals, and eye gaze information, to improve autistic patients' driving abilities.…”

Section: Related Workmentioning

confidence: 99%

MFA: A Smart Glove with Multimodal Intent Sensing Capability

Wang

Feng

Tian

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

At present, virtual-reality fusion smart experiments mostly employ visual perception devices to collect user behavior data, but this method faces the obstacles of distance, angle, occlusion, light, and a variety of other factors of indoor interactive input devices. Moreover, the essence of the traditional multimodal fusion algorithm (TMFA) is to analyze the user’s experimental intent serially using single-mode information, which cannot fully utilize the intent information of each mode. Therefore, this paper designs a multimodal fusion algorithm (hereinafter referred to as MFA, Algorithm 4) which focuses on the parallel fusion of user’s experimental intent. The essence of the MFA is the fusion of multimodal intent probability. At the same time, this paper designs a smart glove based on the virtual-reality fusion experiments, which can integrate multichannel data such as voice, visual, and sensor. This smart glove can not only capture user’s experimental intent but also navigate, guide, or warn user’s operation behaviors, and it has stronger perception capabilities compared to any other data glove or smart experimental device. The experimental results demonstrate that the smart glove presented in this paper can be widely employed in the chemical experiment teaching based on virtual-reality fusion.

show abstract

Section: Related Workmentioning

confidence: 99%

MFA: A Smart Glove with Multimodal Intent Sensing Capability

Wang

Feng

Tian

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…For instance, Ismail, Billinghurst [25] developed a multimodal AR application, and they used both motion and speech recognition for interaction. Chen, Li [26] also developed a mobile AR application for dog training.…”

Section: Related Workmentioning

confidence: 99%

A Smart Multimodal Augmented Reality Application Skill Training For Preoperative Procedures

Taçgin

Taçgın

2020

Bilişim Teknolojileri Dergisi

View full text Add to dashboard Cite

Ameliyat öncesi hazırlıkta cerrahi aletlerin mayo masasına dizilimi için temel kurallar bulunsa da belirli bir standart bulunmamaktadır. Hemşirelik öğrencileri yüksek riskli durumlar ve öğrenmeleri sırasında yetersiz geribildirim alma gibi sınırlılıklar sebebiyle gereken tecrübeyi kazanmak için yeterli olanaklara sahip değildir. Bu çalışma söz konusu bu sorunun çözümü için geliştirilen akili bir çoklu model artırılmış gerçeklik uygulamasını tanıtmaktadır. Sistem uzman cerrahi hemşirelerin dizilimlerini kullanarak oluşturulan 10 doğru ve 10 hatalı dizilimle eğitilmiştir. Bu uygulamayı kullanan öğrenciler ameliyat öncesi hazinlik aşamalarını gerçekleştirdikten sonra kendi dizilimlerinin uygunluğuna ilişkin bireysel geribildirim alabilmektedir. Fiziksel ek hareketleri ile etkileşimin sağlandığı bu uygulama, Meta2 artırılmış gerçeklik gözlüğü kullanılarak geliştirilmiştir. Bu çalışma, geliştirilen çoklu model artırılmış gerçeklik uygulamasının ara yüzü, özellikleri ve akıllı sistem yapısını açıklamaktadır. Anahtar Kelimeler-çoklu model arayüz, artırılmış gerçeklik, hemşirelik eğitimi, akıllı sistemler.

show abstract

“…One of the important aspects is creating appropriate interaction techniques for AR applications that allow end users to interact with virtual content in an intuitive way [12]. In AR, there are several categories of interaction technique and basically, interaction can be divided into three major parts: 1) tangible user interaction, 2) multimodal input [13] and 3) mobile interaction [14]. Since interaction is the wide topic, this paper focuses on handheld mobile interaction and entail 3D interaction in AR due to the rapid advent of the application on cell phones, and now smartphones and tablets, bringing AR almost to the mainstream [15].…”

Section: Handheld Mobile Ar Interfacementioning

confidence: 99%

“…In [63], callout display method is used to display the occluded area near the finger touch but the entire finger below the touch still occluded while Paudisch and Chu [65] suggested new hardware design to perform touch input behind the touchscreen display that not yet been launched in the consumer market. In [99], self-occlusion had been spatially solved by estimating the occluded hand and finger motions and gestures, but it is expensive due to the huge calculation requirement and also the limitation of the numbers of dataset according to FIGURE 13. Some examples of occlusion cases happened in the touch-based and mid-air gestures-based interaction techniques; a virtual object occluded by the user's fingers [60], b indirect touch to avoid occlusion [66], c i-virtual object on screen ii-virtual object occluded by finger [61], d Self-occlusions [91], [94], [96], [97], e Major features of AR marker had been occluded [95].…”

Section: Remaining Issues In 3d Object Manipulation Within Handheld Mobile Armentioning

confidence: 99%

3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review

2019

Self Cite

View full text Add to dashboard Cite

Handheld mobile devices referring to mobile phones and tablets become the major output medium for augmented reality (AR), which have seen significant growth in popularity and usage among the public due to the growing release of consumer-oriented communication products nowadays, especially touchscreen smartphones. Unlike traditional desktop or tabletop AR (large display-based) and head-mounted display-based AR systems, small display-based AR (handheld mobile display) requires different interaction techniques that mostly utilize single-hand interaction as well as the limitation of small screen display and limited activity time due to the battery operation hour of handheld mobile devices. However, in handheld mobile AR, research is still lacking, especially research that focuses on 3D interaction in handheld mobile AR for virtual object manipulation. Thus, this paper provides an overview of 3D interaction techniques in handheld mobile AR with critical analysis. First, we describe three main interaction technique categories that applicable in handheld mobile AR, which is touch-based interaction, mid-air gestures-based interaction, and device-based interaction techniques, of their basic concepts on 3D object manipulation. Then, we classify and systematize the highlighted techniques and discuss the advantages and drawbacks of each. Previous studies for widely used techniques have been studied comprehensively. We then draw up a comparison among the different techniques based on the important elements considered in handheld mobile AR. The aim of this paper is to provide researchers with background information on AR and those who are interested in the field of 3D interaction, realizing each technique category. INDEX TERMS 3D object manipulation, device-based interaction technique, mid-air gestures-based interaction technique, handheld mobile AR, touch-based interaction technique. I. INTRODUCTIONAccording to Milgram and Kishino [1], augmented reality (AR) refers to all cases in which the display of an otherwise real environment is augmented by means of virtual (computer graphic) objects. Extended from this definition, Azuma [2] defined that AR is the variation of virtual environments or virtual reality; however, unlike virtual reality that immerses a user inside a synthetic environment and the user cannot see the real world around him/her, AR allows the user to see the real world, with virtual objects composited with the real world. Beyond that, some researchers have doneThe associate editor coordinating the review of this manuscript and approving it for publication was Waleed Alsabhan. a number of surveys or reviews about AR. For example, Krevelen and Poelman [3], drawing up conclusions from several previous works, defined the AR system as: 1) combining real and virtual objects in a real environment; 2) registering real and virtual objects with each other and 3) running interactively, in 3D and in real-time. From another point of view, Mekni and Lemieux [4] proposed to define AR as systems that have the following charact...

show abstract

Designing an Augmented Reality Multimodal Interface for 6DOF Manipulation Techniques

Cited by 15 publications

References 8 publications

MFA: A Smart Glove with Multimodal Intent Sensing Capability

MFA: A Smart Glove with Multimodal Intent Sensing Capability

A Smart Multimodal Augmented Reality Application Skill Training For Preoperative Procedures

3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review

Contact Info

Product

Resources

About