Interactive Multi-Modal Robot Programming

Iba, Soshi; Paredis, Christiaan J. J.; Khosla, Pradeep K.

doi:10.1007/11552246_48

Cited by 11 publications

(5 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2020), which applied gesture and voice based multi-channel fusion to efficiently complete a man-machine collaborative task via an interaction algorithm. A multimodal interactive robotic framework based on a three-step approach: multimodal recognition, intention interpretation and prioritized task execution was presented in Iba et al (2005). The multimodal recognition module translated hand gestures and spontaneous speech into a structured symbolic data stream without abstracting user intent.…”

Section: Multimodal Interactive Robotmentioning

confidence: 99%

See 1 more Smart Citation

A survey of technologies supporting design of a multimodal interactive robot for military communication

Paul

2023

JDAL

View full text Add to dashboard Cite

PurposeThis paper presents a survey of research into interactive robotic systems for the purpose of identifying the state of the art capabilities as well as the extant gaps in this emerging field. Communication is multimodal. Multimodality is a representation of many modes chosen from rhetorical aspects for its communication potentials. The author seeks to define the available automation capabilities in communication using multimodalities that will support a proposed Interactive Robot System (IRS) as an AI mounted robotic platform to advance the speed and quality of military operational and tactical decision making.Design/methodology/approachThis review will begin by presenting key developments in the robotic interaction field with the objective of identifying essential technological developments that set conditions for robotic platforms to function autonomously. After surveying the key aspects in Human Robot Interaction (HRI), Unmanned Autonomous System (UAS), visualization, Virtual Environment (VE) and prediction, the paper then proceeds to describe the gaps in the application areas that will require extension and integration to enable the prototyping of the IRS. A brief examination of other work in HRI-related fields concludes with a recapitulation of the IRS challenge that will set conditions for future success.FindingsUsing insights from a balanced cross section of sources from the government, academic, and commercial entities that contribute to HRI a multimodal IRS in military communication is introduced. Multimodal IRS (MIRS) in military communication has yet to be deployed.Research limitations/implicationsMultimodal robotic interface for the MIRS is an interdisciplinary endeavour. This is not realistic that one can comprehend all expert and related knowledge and skills to design and develop such multimodal interactive robotic interface. In this brief preliminary survey, the author has discussed extant AI, robotics, NLP, CV, VDM, and VE applications that is directly related to multimodal interaction. Each mode of this multimodal communication is an active research area. Multimodal human/military robot communication is the ultimate goal of this research.Practical implicationsA multimodal autonomous robot in military communication using speech, images, gestures, VST and VE has yet to be deployed. Autonomous multimodal communication is expected to open wider possibilities for all armed forces. Given the density of the land domain, the army is in a position to exploit the opportunities for human–machine teaming (HMT) exposure. Naval and air forces will adopt platform specific suites for specially selected operators to integrate with and leverage this emerging technology. The possession of a flexible communications means that readily adapts to virtual training will enhance planning and mission rehearsals tremendously.Social implicationsInteraction, perception, cognition and visualization based multimodal communication system is yet missing. Options to communicate, express and convey information in HMT setting with multiple options, suggestions and recommendations will certainly enhance military communication, strength, engagement, security, cognition, perception as well as the ability to act confidently for a successful mission.Originality/valueThe objective is to develop a multimodal autonomous interactive robot for military communications. This survey reports the state of the art, what exists and what is missing, what can be done and possibilities of extension that support the military in maintaining effective communication using multimodalities. There are some separate ongoing progresses, such as in machine-enabled speech, image recognition, tracking, visualizations for situational awareness, and virtual environments. At this time, there is no integrated approach for multimodal human robot interaction that proposes a flexible and agile communication. The report briefly introduces the research proposal about multimodal interactive robot in military communication.

show abstract

Section: Multimodal Interactive Robotmentioning

confidence: 99%

“…A multimodal interactive robotic framework based on a three-step approach: multimodal recognition, intention interpretation and prioritized task execution was presented in Iba et al . (2005).…”

Section: Multimodal Interactions and Communicationsmentioning

confidence: 99%

A survey of technologies supporting design of a multimodal interactive robot for military communication

Paul

2023

JDAL

View full text Add to dashboard Cite

show abstract

“…Usually, mechatronic systems use predefined interaction language specifically oriented to a particular application; an interesting experience among them is described by Iba et al [10]. They use multimodal interaction between humans and robots in order to describe a framework to compose robot programs by non expert people.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Language Specification for Human Adaptive Mechatronics

Ferri¹,

D’Ulizia²,

Grifoni

2012

JNIT

View full text Add to dashboard Cite

Designing and building automated systems with which people can interact naturally is one of the emerging objective of Mechatronics. In this perspective multimodality and adaptivity represent focal issues, enabling users to communicate more freely and naturally with automated systems. One of the basic problem of multimodal interaction is the fusion process. Current approaches to fusion are mainly two: the former implements the multimodal fusion at dialogue management level, whereas the latter at grammar level. In this paper, we propose a multimodal attribute grammar, that provides constructions both for representing input symbols from different modalities and for modeling semantic and temporal features of multimodal input symbols, enabling the specification of multimodal languages. Moreover, an application of the proposed approach in the context of a multimodal language specification to control a driver assistance system, as robots using different integrated interaction modalities, is given.Comment: 11 pages, 4 figure

show abstract

“…Perzanowski et al (2001) use a speech, gesture, and graphical interface to study natural interactions between humans and robots. Iba, Paredis, and Khosla (2002) have developed a system that incorporates speech and gesture, instead of a keyboard and joystick, to program a vacuum-cleaning mobile robot. These types of systems target users who are experts at a particular task but may have limited programming ability.…”

Section: Related Workmentioning

confidence: 99%

Predictive Robot Programming: Theoretical and Experimental Analysis

Dixon

Dolan

Khosla

2004

The International Journal of Robotics Research

Self Cite

View full text Add to dashboard Cite

As the capabilities of manipulator robots increase, they are performing more complex tasks. The cumbersome nature of conventional programming methods limits robotic automation due to the lengthy programming time. We present a novel method for reducing the time needed to program a manipulator robot: predictive robot programming (PRP). The PRP system constructs a statistical model of the user by incorporating information from previously completed tasks. Using this model, the PRP system computes predictions about where the user will move the robot. The user can reduce programming time by allowing the PRP system to complete the task automatically. In this paper, we derive a learning algorithm that estimates the structure of continuous-density hidden Markov models from tasks the user has already completed. We analyze the performance of the PRP system on two sets of data. The first set is based on data from complex, realworld robotic tasks. We show that the PRP system is able to compute predictions for about 25% of the waypoints with a median prediction error less than 0.5% of the distance traveled during prediction. We also present laboratory experiments showing that the PRP system results in a significant reduction in programming time, with users completing simple robot-programming tasks over 30% faster when using the PRP system to compute predictions of future positions.

show abstract

Interactive Multi-Modal Robot Programming

Cited by 11 publications

References 66 publications

A survey of technologies supporting design of a multimodal interactive robot for military communication

A survey of technologies supporting design of a multimodal interactive robot for military communication

Multimodal Language Specification for Human Adaptive Mechatronics

Predictive Robot Programming: Theoretical and Experimental Analysis

Contact Info

Product

Resources

About