Branching Gaussian Processes with Applications to Spatiotemporal Reconstruction of 3D Trees

Abstract

We propose a robust method for estimating dynamic 3D curvilinear branching structure from monocular images. While 3D reconstruction from images has been widely studied, estimating thin structure has received less attention. This problem becomes more challenging in the presence of camera error, scene motion, and a constraint that curves are attached in a branching structure. We propose a new generalpurpose prior, a branching Gaussian processes (BGP), that models spatial smoothness and temporal dynamics of curves while enforcing attachment between them. We apply this prior to fit 3D trees directly to image data, using an efficient scheme for approximate inference based on expectation propagation. The BGP prior's Gaussian form allows us to approximately marginalize over 3D trees with a given model structure, enabling principled comparison between tree models with varying complexity. We test our approach on a novel multi-view dataset depicting plants with known 3D structures and topologies undergoing small nonrigid motion. Our method outperforms a state-of-the-art 3D reconstruction method designed for non-moving thin structure. We evaluate under several common measures, and we propose a new measure for reconstructions of branching multi-part 3D scenes under motion.

BGP: Branched Gaussian processes for identifying gene-specific branching dynamics in single cell data

“…Another useful extension would be to jointly infer pseudotime and branching behaviour, which would also improve uncertainty estimation as the uncertainty arising from the estimation of the former would be included in the posterior branching uncertainty. Extending our model to multiple branching points is straightforward from a modelling standpoint but presents a more challenging optimisation problem wherein a tree prior on the branching structure may prove helpful (Simek et al, 2016). This extension would allow us to address the problem of selecting the correct number of branches in the global cellular branching dynamics.…”

Segmentation and 3D reconstruction of rose plants from stereoscopic images

“…For this, they apply a region growing algorithm and fit geometric primitives to the segmented meshes. Simek et al [51] proposed an approach based on temporal information and Gaussian processes to model the branches from multiple 2D images. Similarly, Gélard et al [21] build the 3D model of a plant using structure from motion.…”

“…For the evaluation of the proposed method we used a total of five datasets, two downloaded from the state of the art (called TB-Roses v2 [54] and Arabidopsis [51]), and three datasets created by the authors, which jointly are called ROSeS (Roses for Object Segmentation and Skeletonization) 2 . These three datasets are divided into: S-ROSeS, a synthetic dataset of rose bushes, H-ROSeS, a hybrid dataset with 200 real indoor images but with realistic plastic plants, and R-ROSeS, a completely real dataset with 100 photos of rose bushes taken in different botanic gardens.…”

“…The images are provided together with the ground truth marking the segmented branches. The Arabidopsis dataset [51] consists of 160 images with a resolution of 2208×1656 px of twelve Arabidopsis genus plants taken indoor 5 . The images were taken by rotating a turntable by 10 degrees of yaw.…”