Combining Stereo and Time-of-Flight Images with Application to Automatic Plant Phenotyping

Abstract

Abstract. This paper shows how stereo and Time-of-Flight (ToF) images can be combined to estimate dense depth maps in order to automate plant phenotyping. We focus on some challenging plant images captured in a glasshouse environment, and show that even the state-of-the-art stereo methods produce unsatisfactory results. By developing a geometric approach which transforms depth information in a ToF image to a localised search range for dense stereo, a global optimisation strategy is adopted for producing smooth and discontinuity-preserving results. Since pixel-by-pixel depth data are unavailable for our images and many other applications, a quantitative method accounting for the surface smoothness and the edge sharpness to evaluate estimation results is proposed. We compare our method with and without ToF against other state-ofthe-art stereo methods, and demonstrate that combining stereo and ToF images gives superior results.

A Survey on Time-of-Flight Stereo Fusion

“…They are often based on a line search that is guided by the ToF data. Global methods, [5,31,35,27,44,26,32,51,52,40,45] add the ToF information as an additional data term in a global energy functional is then jointly optimized. While the depth maps obtained are smoother due to the usage of prior information/regularizers, this is at the cost of additional computational resources.…”

“…In this overview, we will further group the global techniques depending the framework that was chosen for optimization. While [31,35,27,44,45] employ different graphical models for inference, [51,32] formulate the problem in a variational framework. The last sub-group of the global methods [5,26,52,40] contains those which use other non-local optimization strategies.…”

“…For many applications geometry reconstruction is not the final goal but just a intermediate step. Song et al [45] evaluate the edge quality by comparing the obtained depth edges with pre-labeled silhouette boundaries in a plant phenotyping application (cf. Chapter 4, Section 5.2 on page 69).…”

Data Fusion of Lidar Into a Region Growing Stereo Algorithm

“…A priori methods using ToFCs have been used with most of the common state-of-the-art stereo matching algorithms including dynamic programming (DP) (Gudmundsson et al, 2008), graph cuts (Hahne and Alexa, 2008); (Song et al, 2011), belief propagation (Jiejie Zhu et al, 2011) and semi-global matching (SGM) (Fischer et al, 2011). ToF range data is used to overcome the limitations of stereo in homogenous image regions while stereo range data is retained near depth discontinuities.…”

A Game Engine as a Generic Platform for Real-Time Previz-on-Set in Cinema Visual Effects

“…For instance [3] uses a trifocal rig to generate high precision depth maps: the authors combine a stereo system with a monocular depth sensor, similar to [29,12,25,8]. Then, a complex global optimization workflow is needed to merge data.…”