GhostAR: A Time-space Editor for Embodied Authoring of Human-Robot Collaborative Task with Augmented Reality

Cao, Yichao; Wang, Tianyi; Qian, Xun; Rao, Pawan S.; Wadhawan, Manav; Huang, Ke; Ramani, Karthik

doi:10.1145/3332165.3347902

Cited by 61 publications

(46 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A spatial programming by demonstration (PBD) called GhostAR was developed in Cao et al (2019) , which captures the real-time motion of the human, feeds it to a dynamic time warping (DTW) algorithm which maps it to an authored human motion, and outputs corresponding robot actions in a human-lead robot-assist scenario. The captured human motions and the corresponding robot actions are saved and visualized to the user who can observe the complete demonstration with saved AR ghosts of both the human and robot and interactively perform edits on robot actions to clarify user intent.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Bassyouni

Elhajj

2021

Front. Robot. AI

View full text Add to dashboard Cite

Recently, advancements in computational machinery have facilitated the integration of artificial intelligence (AI) to almost every field and industry. This fast-paced development in AI and sensing technologies have stirred an evolution in the realm of robotics. Concurrently, augmented reality (AR) applications are providing solutions to a myriad of robotics applications, such as demystifying robot motion intent and supporting intuitive control and feedback. In this paper, research papers combining the potentials of AI and AR in robotics over the last decade are presented and systematically reviewed. Four sources for data collection were utilized: Google Scholar, Scopus database, the International Conference on Robotics and Automation 2020 proceedings, and the references and citations of all identified papers. A total of 29 papers were analyzed from two perspectives: a theme-based perspective showcasing the relation between AR and AI, and an application-based analysis highlighting how the robotics application was affected. These two sections are further categorized based on the type of robotics platform and the type of robotics application, respectively. We analyze the work done and highlight some of the prevailing limitations hindering the field. Results also explain how AR and AI can be combined to solve the model-mismatch paradigm by creating a closed feedback loop between the user and the robot. This forms a solid base for increasing the efficiency of the robotic application and enhancing the user’s situational awareness, safety, and acceptance of AI robots. Our findings affirm the promising future for robust integration of AR and AI in numerous robotic applications.

show abstract

Section: Discussionmentioning

confidence: 99%

“…In Cao et al (2019) , the human motion is captured through the AR elements (Oculus Rift and two Oculus Touch Controllers) and saved as ghost holograms. Dynamic Time Warping is used to infer the human motion in real time from a previously compiled list of groups that represent human authorized motions.…”

Section: Discussionmentioning

confidence: 99%

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Bassyouni

Elhajj

2021

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…Buildings recede as you walk 'back' through time, forests wind back fire damage, glaciers melt or rise as carbon-breathing humanity approaches and so on. Past thought experiments, nascent experiments in AR interfaces of time explore 'dynamic time warping' (DTW) (Cao et al, 2019) that combines human interaction with repeatable robot motion to achieve 'adaptive collaboration' exist.…”

Section: Four-dimensional Place(ment)mentioning

confidence: 99%

Conceptualising Augmented Reality: From virtual divides to mediated dynamics

Heemsbergen

Bowtell

Vincent

2021

Convergence: The International Journal of Research into New Med

View full text Add to dashboard Cite

Although Augmented Reality (AR) scholarship is largely defined through technocentric boundary work that delineates the virtual from the real, it is nevertheless vital to consider experiential conceptualisations of AR as mediating the human–physical environment, as this makes visible for analysis specific properties that afford specific dynamics of augmented publics. We consider how AR mediates the environment in ways previous media could not, identifying four affordances of note. We name visual (dis)integrity, environmental activation, contextual pointalisation and four-dimensional place(ment), as well as reliance on digital infrastructures as sets of properties and dynamics that speak to what AR affords its users. The article first traces how the conceptualisation of AR in scholarship has yet to move past technocentric metaphors of description, adopted from the Virtuality Continuum (Milgram et al., 1995) that separates reality from mediating technologies. It then pushes media critique conceptualising AR in ways that more accurately account for lived experiences of perception. Doing so updates the metaphors used to understand AR and exposes first- and second-order affordances of AR media, which define but are not definitive of the constraints and potentials present in how the properties and dynamics of AR mediate perceptions of life. The article concludes by noting how future research can now consider definitions of AR media that centre on how perceptible spatial computation augments relations between objects, whether these are conjured from electrons, atoms or humans.

show abstract

“…When the desired requirements are achieved, the authoring process is finished and the robot is ready to be deployed to interact autonomously. Authoring differs from classic programming in its focus on end users with limited background in computer sciences and seeks to address questions of how can these users design, or author, behaviors using modalities such as tangible interactions ( Sefidgar et al, 2017 ; Huang and Cakmak, 2017 ), natural language ( Walker et al, 2019 ), augmented- or mixed-reality ( Cao et al, 2019a ; Peng et al, 2018 ; Akan et al, 2011 ; Gao and Huang, 2019 ), visual programming environments ( Glas et al, 2016 ; Paxton et al, 2017 ), or a mixture of modalities ( Huang and Cakmak, 2017 ; Porfirio et al, 2019 ). Steinmetz et al (2018) describe task-level programming as parameterizing and sequencing predefined skills composed of primitives to solve a task at hand.…”

Section: Related Workmentioning

confidence: 99%

Task-Level Authoring for Remote Robot Teleoperation

et al. 2021

View full text Add to dashboard Cite

Remote teleoperation of robots can broaden the reach of domain specialists across a wide range of industries such as home maintenance, health care, light manufacturing, and construction. However, current direct control methods are impractical, and existing tools for programming robot remotely have focused on users with significant robotic experience. Extending robot remote programming to end users, i.e., users who are experts in a domain but novices in robotics, requires tools that balance the rich features necessary for complex teleoperation tasks with ease of use. The primary challenge to usability is that novice users are unable to specify complete and robust task plans to allow a robot to perform duties autonomously, particularly in highly variable environments. Our solution is to allow operators to specify shorter sequences of high-level commands, which we call task-level authoring, to create periods of variable robot autonomy. This approach allows inexperienced users to create robot behaviors in uncertain environments by interleaving exploration, specification of behaviors, and execution as separate steps. End users are able to break down the specification of tasks and adapt to the current needs of the interaction and environments, combining the reactivity of direct control to asynchronous operation. In this paper, we describe a prototype system contextualized in light manufacturing and its empirical validation in a user study where 18 participants with some programming experience were able to perform a variety of complex telemanipulation tasks with little training. Our results show that our approach allowed users to create flexible periods of autonomy and solve rich manipulation tasks. Furthermore, participants significantly preferred our system over comparative more direct interfaces, demonstrating the potential of our approach for enabling end users to effectively perform remote robot programming.

show abstract

GhostAR: A Time-space Editor for Embodied Authoring of Human-Robot Collaborative Task with Augmented Reality

Cited by 61 publications

References 30 publications

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Conceptualising Augmented Reality: From virtual divides to mediated dynamics

Task-Level Authoring for Remote Robot Teleoperation

Contact Info

Product

Resources

About