RayTran: 3D Pose Estimation and Shape Reconstruction of Multiple Objects from Videos with Ray-Traced Transformers

Tyszkiewicz, Michał J.; Maninis, Kevis-Kokitsi; Popov, Stefan; Ferrari, Vittorio

doi:10.1007/978-3-031-20080-9_13

Cited by 12 publications

(4 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They are either trained on synthetic data [36], or on small real datasets [18,23,24]. Similarly, recent learning-based approaches reconstruct a scene from a video [30,25,26,49,42], and use Scan2CAD as their main evaluation benchmark. Our CAD-Estate dataset can benefit all of these works as it offers new, large-scale, diverse, real video data with annotated complex spatial arrangements of 3D objects into scenes.…”

Section: Related Workmentioning

confidence: 99%

“…The final goal is to detect all objects in the scene, recognize their class, reconstruct their 3D shape, as well as their pose within the overall scene coordinate frame. With the advances of scalable deep learning techniques, the field has progressed from reconstructing the 3D shape of one object in a simple image with trivial background [32,50,40,33,8,17], to limited reasoning about object arrangements in simple multi-object scenes [36,18,23], and finally to unrestricted multi-object 3D reconstruction in complex real-world scenes [49,30,42,12]. This evolution has been dependent on the availability of ever larger and more diverse data sets for training and evaluation [3,10,6,44,47,7,27,15,16] Existing datasets for Semantic 3D scene understanding fall broadly in two categories: synthetic and acquired from real images/videos.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos

Maninis

Popov

Niesser

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos

Maninis

Popov

Niesser

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…[4,7,32,34,35,47,55,56] are very similar to these approach with slight variations, all basically doing direct pose regression from a CNN. There are many other examples which use combinations of these approaches and add in depth data [17,22,41,[48][49][50][51], however depth data is outside the scope of the desired solution here.…”

Section: Other Pose Detectorsmentioning

confidence: 99%

An analysis of precision: occlusion and perspective geometry’s role in 6D pose estimation

Choate,

Worth,

Nykl

et al. 2023

Neural Comput & Applic

View full text Add to dashboard Cite

Achieving precise 6 degrees of freedom (6D) pose estimation of rigid objects from color images is a critical challenge with wide-ranging applications in robotics and close-contact aircraft operations. This study investigates key techniques in the application of YOLOv5 object detection convolutional neural network (CNN) for 6D pose localization of aircraft using only color imagery. Traditional object detection labeling methods suffer from inaccuracies due to perspective geometry and being limited to visible key points. This research demonstrates that with precise labeling, a CNN can predict object features with near-pixel accuracy, effectively learning the distinct appearance of the object due to perspective distortion with a pinhole camera. Additionally, we highlight the crucial role of knowledge about occluded features. Training the CNN with such knowledge slightly reduces pixel precision, but enables the prediction of 3 times more features, including those that are not initially visible, resulting in an overall better performing 6D system. Notably, we reveal that the data augmentation technique of scale can interfere with pixel precision when used during training. These findings are crucial for the entire system, which leverages the Solve Perspective-N-Point (Solve-PnP) algorithm, achieving 6D pose accuracy within 1$$^\circ$$ ∘ and 7 cm at distances ranging from 7.5 to 35 m from the camera. Moreover, this solution operates in real-time, achieving sub-10ms processing times on a desktop PC.

show abstract

“…This lets us avoid complex non-linear optimization and makes the joint estimation amenable to potential on-board implementation. Most important, it lets us fully leverage the structured bias underlying category-level object pose estimation -Scan2CAD [4] -CAD models -RayTran [34] -Voxel Occupancy -Vid2CAD [24] CAD models Moving Avg.…”

Section: Introductionmentioning

confidence: 99%

TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation

Yoshitake¹,

Nishimura²,

Nobuhara³

et al. 2023

Preprint

View full text Add to dashboard Cite

We propose a novel method for joint estimation of shape and pose of rigid objects from their sequentially observed RGB-D images. In sharp contrast to past approaches that rely on complex non-linear optimization, we propose to formulate it as a neural optimization that learns to efficiently estimate the shape and pose. We introduce Deep Directional Distance Function (DeepDDF), a neural network that directly outputs the depth image of an object given the camera viewpoint and viewing direction, for efficient error computation in 2D image space. We formulate the joint estimation itself as a Transformer which we refer to as Trans-Poser. We fully leverage the tokenization and multi-head attention to sequentially process the growing set of observations and to efficiently update the shape and pose with a learned momentum, respectively. Experimental results on synthetic and real data show that DeepDDF achieves high accuracy as a category-level object shape representation and TransPoser achieves state-of-the-art accuracy efficiently for joint shape and pose estimation.

show abstract

RayTran: 3D Pose Estimation and Shape Reconstruction of Multiple Objects from Videos with Ray-Traced Transformers

Cited by 12 publications

References 55 publications

Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos

Vid2CAD: CAD Model Alignment Using Multi-View Constraints From Videos

An analysis of precision: occlusion and perspective geometry’s role in 6D pose estimation

TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation

Contact Info

Product

Resources

About