Learning Spatial Common Sense With Geometry-Aware Recurrent Networks

Tung, Hsiao-Yu Fish; Cheng, Ricson; Fragkiadaki, Katerina

doi:10.1109/cvpr.2019.00270

Cited by 56 publications

(57 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Policies have to be learned to aid agents move around a scene and this is the task of active recognition [45,46,47,48,49]). The policy will be learned at the same time it is learning other tasks and representation and it will tell the agent where and how to strategically move to recognize things faster [50,51].…”

Section: Embodied Visual Recognitionmentioning

confidence: 99%

Embodied AI-Driven Operation of Smart Cities: A Concise Review

SHENAVARMASOULEH¹,

Mohammadi²,

Amini³

et al. 2021

Preprint

View full text Add to dashboard Cite

<div>A smart city can be seen as a framework, comprised of Information and Communication Technologies (ICT). An intelligent network of connected devices that collect data with their sensors and transmit them using wireless and cloud technologies in order to communicate with other assets in the ecosystem plays a pivotal role in this framework. Maximizing the quality of life of citizens, making better use of available resources, cutting costs, and improving sustainability are the ultimate goals that a smart city is after. Hence, data collected from these connected devices will continuously get thoroughly analyzed to gain better insights into the services that are being offered across the city; with this goal in mind that they can be used to make the whole system more efficient.</div><div>Robots and physical machines are inseparable parts of a smart city. Embodied AI is the field of study that takes a deeper look into these and explores how they can fit into real-world environments. It focuses on learning through interaction with the surrounding environment, as opposed to Internet AI which tries to learn from static datasets. Embodied AI aims to train an agent that can See (Computer Vision), Talk (NLP), Navigate and Interact with its environment (Reinforcement Learning), and Reason (General Intelligence), all at the same time. Autonomous driving cars and personal companions are some of the examples that benefit from Embodied AI nowadays.</div><div>In this paper, we attempt to do a concise review of this field. We will go through its definitions, its characteristics, and its current achievements along with different algorithms, approaches, and solutions that are being used in different components of it (e.g. Vision, NLP, RL). We will then explore all the available simulators and 3D interactable databases that will make the research in this area feasible. Finally, we will address its challenges and identify its potentials for future research.</div>

show abstract

Section: Embodied Visual Recognitionmentioning

confidence: 99%

Embodied AI-Driven Operation of Smart Cities: A Concise Review

SHENAVARMASOULEH¹,

Mohammadi²,

Amini³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…They are designed for geometric completeness given their dense grid-like structure. Many previous works have demonstrated the strength of such representations in geometry modeling [8,9,15,61,62,67,78]. Some recent works [27,45,54] have also shown their effectiveness in modeling appearance.…”

Section: Related Workmentioning

confidence: 99%

HVH: Learning a Hybrid Neural Volumetric Representation for Dynamic Hair Performance Capture

Wang¹,

Nam²,

Stuyck³

et al. 2021

Preprint

View full text Add to dashboard Cite

Capturing and rendering life-like hair is particularly challenging due to its fine geometric structure, the complex physical interaction and its non-trivial visual appearance. Yet, hair is a critical component for believable avatars. In this paper, we address the aforementioned problems: 1) we use a novel, volumetric hair representation that is composed of thousands of primitives. Each primitive can be rendered efficiently, yet realistically, by building on the latest advances in neural rendering. 2) To have a reliable control signal, we present a novel way of tracking hair on the strand level. To keep the computational effort manageable, we use guide hairs and classic techniques to expand those into a dense hood of hair. 3) To better enforce temporal consistency and generalization ability of our model, we further optimize the 3D scene flow of our representation with multiview optical flow, using volumetric raymarching. Our method can not only create realistic renders of recorded multi-view sequences, but also create renderings for new hair configurations by providing new control signals. We compare our method with existing work on viewpoint synthesis and drivable animation and achieve stateof-the-art results. Please check out our project website at https://ziyanw1.github.io/hvh/.

show abstract

“…However, it requires camera poses computed from SfM for training, and point cloud estimation can be inaccurate when the test image is out of distribution. Instead of using point clouds, neural 3D representations, including implicit function [43,59] and the deep voxels [13,19,58,47,46] have shown impressive reconstruction and synthesis results. Our work is closely related to the approach proposed by Tung et al [13], which leverages view prediction for learning latent 3D voxel structure of the scene.…”

Section: Related Workmentioning

confidence: 99%

“…Instead of using point clouds, neural 3D representations, including implicit function [43,59] and the deep voxels [13,19,58,47,46] have shown impressive reconstruction and synthesis results. Our work is closely related to the approach proposed by Tung et al [13], which leverages view prediction for learning latent 3D voxel structure of the scene. However, camera pose is still required to provide supervision.…”

Section: Related Workmentioning

confidence: 99%

“…Our work is also highly inspired by Nguyen-Phuoc et al [47], who proposed inserting the voxel representation into Generative Adversarial Networks, enabling the disentanglement of 3D structure, style, and pose. Finally, our work is related to plenty of downstream tasks that leverages a learned 3D deep voxels, such as 3D object detection [13], 3D object tracking [18], 3D motion estimation [19] and few-shot concept learning [52].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Video Autoencoder: self-supervised disentanglement of static 3D structure and motion

Lai

Liu

Efros

et al. 2021

Preprint

View full text Add to dashboard Cite

Videostructure + motion … …Figure 1: Video Autoencoder: raw input video is automatically disentangled into 3D scene structure and camera trajectory. To reconstruct the original video, the camera transformation is applied to the 3D structure feature and then decoded back to pixels. Without any fine-tuning, the model generalizes to unseen videos and enables tasks such as novel view synthesis, pose estimation, and "video following".

show abstract

Learning Spatial Common Sense With Geometry-Aware Recurrent Networks

Cited by 56 publications

References 17 publications

Embodied AI-Driven Operation of Smart Cities: A Concise Review

Embodied AI-Driven Operation of Smart Cities: A Concise Review

HVH: Learning a Hybrid Neural Volumetric Representation for Dynamic Hair Performance Capture

Video Autoencoder: self-supervised disentanglement of static 3D structure and motion

Contact Info

Product

Resources

About