Actionet: An Interactive End-To-End Platform For Task-Based Data Collection And Augmentation In 3D Environment

Duan, Jiafei; Jian, Samson Yu Bai; Tan, Hui Li; Tan, Cheston

doi:10.1109/icip40778.2020.9191324

Cited by 8 publications

(7 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Datasets. We evaluate PlaTe on an instructional video dataset CrossTask [16] and an interactive dataset ActioNet [30], which is based on AI2-THOR [31]. For real-world UR-5 experiments, we collect a UR-5 Reaching Dataset which consists of 100 trajectories (2150 first-person-view RGB image and corresponding action pairs) as a training set and evaluate on a real UR-5 platform.…”

Section: A Experimental Setupmentioning

confidence: 99%

See 1 more Smart Citation

PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks

Sun,

Huang,

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we study the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the ability to plan a goal-directed procedure from real-world videos. Learning structured and plannable state and action spaces directly from unstructured videos is the key technical challenge of our task. There are two problems: first, the appearance gap between the training and validation datasets could be large for unstructured videos; second, these gaps lead to decision errors that compound over the steps. We address these limitations with Planning Transformer (PlaTe), which has the advantage of circumventing the compounding prediction errors that occur with single-step models during long model-based rollouts. Our method simultaneously learns the latent state and action information of assigned tasks and the representations of the decision-making process from human demonstrations. Experiments conducted on real-world instructional videos and an interactive environment show that our method can achieve a better performance in reaching the indicated goal than previous algorithms. We also validated the possibility of applying procedural tasks on a UR-5 platform. Please see pair.toronto.edu/plate-planner for additional details.

show abstract

Section: A Experimental Setupmentioning

confidence: 99%

“…To further illustrate the effectiveness of our method, we experiment on a second dataset ActioNet [30]. ActioNet is an interactive end-to-end platform for data collection and augmentation of the task-based dataset in 3D environment.…”

Section: Evaluating Procedures Planning On Actionetmentioning

confidence: 99%

PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks

Sun,

Huang,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…But VLN sequences are much longer and require a constant feeding input of vision data and the ability to manipulate camera viewpoints, which is unlike VQA which takes in a single input question and performs a series of actions to determine the answer to the question.The notion that we might be able to give out a general, natural language instruction to a robot and expect them to execute or perform the task is now possible. These are [100,10,11] achieved with the advancedment of recurrent neural network methods for joint interpretation of both visual and natural language input and datasets that are designed for simplifying processes of task-based instruction in navigation and performing of tasks in the 3D environment.…”

Section: Types Of Visual Navigationmentioning

confidence: 99%

“…These simulated worlds serve as virtual testbeds to train and test embodied AI frameworks before deploying them into the real world. These embodied AI simulators also facilitate the collection of of task-based dataset [10,11] which are tedious to collect in real-world as it requires an extensive amount of manual labour to replicate the same setting as in the virtual world. While there have been several survey papers in the field of embodied AI [1,12,2], they are mostly outdated as they were published before the modern deep learning era, which started around 2009 [13,14,15,16,8].…”

Section: Introductionmentioning

confidence: 99%

A Survey of Embodied AI: From Simulators to Research Tasks

Duan¹,

Jian²,

Tan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

There has been an emerging paradigm shift from the era of "internet AI" to "embodied AI", whereby AI algorithms and agents no longer simply learn from datasets of images, videos or text curated primarily from the internet. Instead, they learn through embodied physical interactions with their environments, whether real or simulated. Consequently, there has been substantial growth in the demand for embodied AI simulators to support a diversity of embodied AI research tasks. This growing interest in embodied AI is beneficial to the greater pursuit of artificial general intelligence, but there is no contemporary and comprehensive survey of this field. This paper comprehensively surveys state-of-the-art embodied AI simulators and research, mapping connections between these. By benchmarking nine state-of-the-art embodied AI simulators in terms of seven features, this paper aims to understand the simulators in their provision for use in embodied AI research. Finally, based upon the simulators and a pyramidal hierarchy of embodied AI research tasks, this paper surveys the main research tasks in embodied AIvisual exploration, visual navigation and embodied question answering (QA), covering the state-of-the-art approaches, evaluation and datasets.

show abstract

“…However, those works primarily focused on human intuitions of recognizing or predicting motions. But with the advancement and rise of deep learning, computer graphics, and embodied AI [22,23,24], there has been a paradigm shift towards generating synthetic datasets that range from simple 2D cartoons [25,26,27] to realistic interaction in 3D environments [13,28,14,29,30,15], all with the aim to explore machine perception of physics and causal reasoning on a deeper level. However, only datasets from CLEVRER [15], CoPhy [13], and CATER [14] are most relevant to our work.…”

Section: Related Workmentioning

confidence: 99%

SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments

Duan¹,

Jian²,

Tan³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent advancements in deep learning, computer vision, and embodied AI have given rise to synthetic causal reasoning video datasets. These datasets facilitate the development of AI algorithms that can reason about physical interactions between objects. However, datasets thus far have primarily focused on elementary physical events such as rolling or falling. There is currently a scarcity of datasets that focus on the physical interactions that humans perform daily with objects in the real world. To address this scarcity, we introduce SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments. The SPACE simulator allows us to generate the SPACE dataset, a synthetic video dataset in a 3D environment, to systematically evaluate physics-based models on a range of physical causal reasoning tasks. Inspired by daily object interactions, the SPACE dataset comprises videos depicting three types of physical events: containment, stability and contact. These events make up the vast majority of the basic physical interactions between objects. We then further evaluate it with a state-of-the-art physics-based deep model and show that the SPACE dataset improves the learning of intuitive physics with an approach inspired by curriculum learning. Repository: https://github.com/ jiafei1224/SPACE

show abstract

Actionet: An Interactive End-To-End Platform For Task-Based Data Collection And Augmentation In 3D Environment

Cited by 8 publications

References 6 publications

PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks

PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks

A Survey of Embodied AI: From Simulators to Research Tasks

SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments

Contact Info

Product

Resources

About