Structured Labels in Random Forests for Semantic Labelling and Object Detection

Abstract

Ensembles of randomized decision trees, known as Random Forests, have become a valuable machine learning tool for addressing many computer vision problems. Despite their popularity, few works have tried to exploit contextual and structural information in random forests in order to improve their performance. In this paper, we propose a simple and effective way to integrate contextual information in random forests, which is typically reflected in the structured output space of complex problems like semantic image labelling. Our paper has several contributions: We show how random forests can be augmented with structured label information and be used to deliver structured low-level predictions. The learning task is carried out by employing a novel split function evaluation criterion that exploits the joint distribution observed in the structured label space. This allows the forest to learn typical label transitions between object classes and avoid locally implausible label configurations. We provide two approaches for integrating the structured output predictions obtained at a local level from the forest into a concise, global, semantic labelling. We integrate our new ideas also in the Hough-forest framework with the view of exploiting contextual information at the classification level to improve the performance on the task of object detection. Finally, we provide experimental evidence for the effectiveness of our approach on different tasks: Semantic image labelling on the challenging MSRCv2 and CamVid databases, reconstruction of occluded handwritten Chinese characters on the Kaist database and pedestrian detection on the TU Darmstadt databases.

Facade Segmentation With a Structured Random Forest

“…In this section we give an overview of Decision Trees and the Structured Random Forest (Kontschieder et al, 2011a, Kontschieder et al, 2014. The Structured Random Forest is employed to classify each pixel by using the region around it considering its local structure.…”

“…To overcome this drawback, various methods have been proposed for the test and training function selection, such as Principal Component Analysis (PCA) and probabilistic approaches (Kontschieder et al, 2014). In our approach the output of a leaf node is computed as a joint probability distribution of the labels assigned to the leaf node.…”

“…The best split is chosen by computing the information gain of Sj. This continues recursively until the leaf node is reached (Kontschieder et al, 2011a, Kontschieder et al, 2014, Nowozin and Lampert, 2011.…”

The 3D-Pitoti Dataset

“…In our case the segmentation task is a pixelwise binary problem and, hence, the evaluation is based on the predicted segmentation mask and the ground truth mask. Based on these masks we compute the Jaccard index [16], also often termed region based intersection over union (IU ), for which we compute the average over classes (mIU ) as in [17]- [20], the pixel accuracy (PA) [20], [21], the dice similarity coefficient (DSC) [14], the hit rate (HR) [14], [20] and the false acceptance rate (F AR) [14].…”

Nonlinear Regression of Saliency Guided Proposals for Unsupervised Segmentation of Dynamic Scenes

“…We train a random forest similar to the randomized tree algorithms in [26], [27]. Each tree tr in a forest F is trained independently on a random subset of the training set D ⊆ X×H according to the saliency guided foreground hypothesis.…”