Deep Hierarchical Semantic Segmentation

Abstract

Humans are able to recognize structured relations in observation, allowing us to decompose complex scenes into simpler parts and abstract the visual world in multiple levels. However, such hierarchical reasoning ability of human perception remains largely unexplored in current literature of semantic segmentation. Existing work is often aware of flatten labels and predicts target classes exclusively for each pixel. In this paper, we instead address hierarchical semantic segmentation (HSS), which aims at structured, pixel-wise description of visual observation in terms of a class hierarchy. We devise HSSN, a general HSS framework that tackles two critical issues in this task: i) how to efficiently adapt existing hierarchy-agnostic segmentation networks to the HSS setting, and ii) how to leverage the hierarchy information to regularize HSS network learning. To address i), HSSN directly casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. To solve ii), HSSN first explores inherent properties of the hierarchy as a training objective, which enforces segmentation predictions to obey the hierarchy structure. Further, with hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space, so as to generate well-structured pixel representations and improve segmentation eventually. We conduct experiments on four semantic segmentation datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, and PASCAL-Person-Part), with different class hierarchies, segmentation network architectures and backbones, showing the generalization and superiority of HSSN.

Creating a computer assisted ICD coding system: performance metric choice and use of the ICD hierarchy

“…In evaluating the explicit use of the ICD hierarchy (Section III-D), our results indicate that HMC methods did not yield improvements.This contrasts with reports of substantial gains using HMC methods with neural networks [33,34], in particular for ICD classification using medication data [20]. For the latter, we found that our model's classification performance was generally superior to results reported by Hansen et al [20] , as discussed in the next subsection.…”

“…In [S9] introduce the TreeMin loss: This loss can be intuitively understood as always penalizing according to the worst relevant prediction in the hierarchy. One advantage of these logical constraint approaches is that they do not require to introduce new hyperparameters to be tuned.…”

“…In our study, we implemented several of them, employing different approaches to incorporate hierarchical knowledge. These include the use of simple logical rules (roll-down and roll-up) to enforce hierarchical consistency of predictions, modifications of the cost functions based on logical rules such as TreeMin loss [33] and MCLoss [34], as well as hierarchical regularization techniques [35] (Appendix F).…”

MResTNet: A Multi-Resolution Transformer Framework with CNN Extensions for Semantic Segmentation

“…InternImage [ 4 ] is a large-scale CNN that incorporates deformable convolution [ 5 ] variants that allow for capturing long-range dependencies with adaptive spatial aggregation. The HSSN framework, introduced in [ 6 ], adapts existing segmentation networks to incorporate hierarchy information for improved network learning. It forms a tree-structured hierarchy of the latent class dependencies, representing concepts and relationships, which is used in training for mapping pixels and their classes and in inference for finding the best path from root to leaf in the class hierarchy.…”