Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects

Noguchi, Atsuhiro; Iqbal, Umar; Tremblay, Jonathan; Harada, Tatsuya; Gallo, Orazio

doi:10.1109/cvpr52688.2022.00366

Cited by 23 publications

(7 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, MovingParts [YZH*24] studies self‐supervised parts discovery by a motion‐based grouping mechanism that uses slot‐based attention, assuming each group follows a rigid motion. Watch‐it‐move [NIT*22] is the first approach that learns re‐poseable part decomposition from multi‐view videos and foreground masks without any prior knowledge of the structure. Skeletonization of these part‐level decomposition methods enables object re‐posing, as discussed in Sec.…”

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision. It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications. It also facilitates the content creation necessary in computer games and AR/VR by avoiding laborious manual design processes. Further, such models are fundamental for intelligent computing systems that need to interpret real‐world scenes and actions to act and interact safely with the human world. Notably, the world surrounding us is dynamic, and reconstructing models of dynamic, non‐rigidly moving scenes is a severely underconstrained and challenging problem. This state‐of‐the‐art report (STAR) offers the reader a comprehensive summary of state‐of‐the‐art techniques with monocular and multi‐view inputs such as data from RGB and RGB‐D sensors, among others, conveying an understanding of different approaches, their potential applications, and promising further research directions. The report covers 3D reconstruction of general non‐rigid scenes and further addresses the techniques for scene decomposition, editing and controlling, and generalizable and generative modeling. More specifically, we first review the common and fundamental concepts necessary to understand and navigate the field and then discuss the state‐of‐the‐art techniques by reviewing recent approaches that use traditional and machine‐learning‐based neural representations, including a discussion on the newly enabled applications. The STAR is concluded with a discussion of the remaining limitations and open challenges.

show abstract

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

Recent Trends in 3D Reconstruction of General Non‐Rigid Scenes

Yunus,

Lenssen,

Niemeyer

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…There are a number of works that aim to recover 3D keypoints using self-supervised geometric reasoning [12,22], but they are limited to rigid objects. More recent unsupervised methods work for articulated objects from monocular RGB data [9,10,10,18,20,24], multi-view data [16], or point clouds [27], where authors suggest to condition on the predicted keypoints and train a conditional generative model to supervise the keypoints through reconstruction losses. We propose a simpler pipeline where we apply our novel unsupervised losses to the predicted keypoints directly and do not require additional models besides the keypoint predictor itself.…”

Section: Unsupervised Keypoint Localizationmentioning

confidence: 99%

RSF: Optimizing Rigid Scene Flow From 3D Point Clouds Without Labels

Deng

Zakhor

2023

2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Training a 3D human keypoint detector from point clouds in a supervised manner requires large volumes of high quality labels. While it is relatively easy to capture large amounts of human point clouds, annotating 3D keypoints is expensive, subjective, error prone and especially difficult for long-tail cases (pedestrians with rare poses, scooterists, etc.). In this work, we propose GC-KPL -Geometry Consistency inspired Key Point Leaning, an approach for learning 3D human joint locations from point clouds without human labels. We achieve this by our novel unsupervised loss formulations that account for the structure and movement of the human body. We show that by training on a large training set from Waymo Open Dataset [21] without any human annotated keypoints, we are able to achieve reasonable performance as compared to the fully supervised approach. Further, the backbone benefits from the unsupervised training and is useful in downstream fewshot learning of keypoints, where fine-tuning on only 10 percent of the labeled training data gives comparable performance to fine-tuning on the entire set. We demonstrated that GC-KPL outperforms by a large margin over SoTA when trained on entire dataset and efficiently leverages large volumes of unlabeled data.* Work done as an intern at Waymo.

show abstract

“…One downside of general reconstruction methods is that they do not leverage the rigidity and kinematic constraints of articulated objects. To explicitly use such priors, [26] propose a method that processes a multi-view sequence of a moving object and discovers its parts as well as their respective reconstruction and kinematic structure in an unsupervised way. Going one step further, [36] learn a shape and appearance prior for each category which allows them to model accurate reconstructions of articulated objects with only 6 given views.…”

Section: Related Workmentioning

confidence: 99%

“…However, there are a wide variety of real-world objects that do not have a constant shape but can be articulated according to the object's underlying kinematics. There has been great progress in articulated object tracking [5,9,32,37] and reconstruction [10,26] from a sequence of observations. However, a sequence of observations is cumbersome since it often requires prior interaction with the environment.…”

Section: Introductionmentioning

confidence: 99%