Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing

Zhou, Tianfei; Yang, Yi; Liu, Si; Gool, Luc Van

doi:10.1109/cvpr46437.2021.00167

Cited by 75 publications

(24 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the deep learning era, human parsing, as a sub-field of scene parsing, became active. Some recent human parsers explored human part relations, based on the human hierarchy [36,56,80,83,110,111]. Only very few efforts [43,45,89,104] are concerned with utilizing structured knowledge to aid the training of general-purpose semantic segmentation networks.…”

Section: Related Workmentioning

confidence: 99%

Deep Hierarchical Semantic Segmentation

Li¹,

Zhou²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Humans are able to recognize structured relations in observation, allowing us to decompose complex scenes into simpler parts and abstract the visual world in multiple levels. However, such hierarchical reasoning ability of human perception remains largely unexplored in current literature of semantic segmentation. Existing work is often aware of flatten labels and predicts target classes exclusively for each pixel. In this paper, we instead address hierarchical semantic segmentation (HSS), which aims at structured, pixel-wise description of visual observation in terms of a class hierarchy. We devise HSSN, a general HSS framework that tackles two critical issues in this task: i) how to efficiently adapt existing hierarchy-agnostic segmentation networks to the HSS setting, and ii) how to leverage the hierarchy information to regularize HSS network learning. To address i), HSSN directly casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. To solve ii), HSSN first explores inherent properties of the hierarchy as a training objective, which enforces segmentation predictions to obey the hierarchy structure. Further, with hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space, so as to generate well-structured pixel representations and improve segmentation eventually. We conduct experiments on four semantic segmentation datasets (i.e., Mapillary Vistas 2.0, Cityscapes, LIP, and PASCAL-Person-Part), with different class hierarchies, segmentation network architectures and backbones, showing the generalization and superiority of HSSN.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Hierarchical Semantic Segmentation

Li¹,

Zhou²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Fully convolutional network (FCN) [38] is selected as the basic structure of our network since the grasp label is pixellevel. FCN is widely used for semantic segmentation of images [39][40][41][42]. Our previous work [25] prove that FCN is effective for predicting dense grasp poses.…”

Section: A Network Architecturementioning

confidence: 88%

On-Policy Pixel-Level Grasping Across the Gap Between Simulation and Reality

Wang¹,

Chang²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Grasp detection in cluttered scenes is a very challenging task for robots. Generating synthetic grasping data is a popular way to train and test grasp methods, as is Dex-net and GraspNet; yet, these methods generate training grasps on 3D synthetic object models, but evaluate at images or point clouds with different distributions, which reduces performance on real scenes due to sparse grasp labels and covariate shift. To solve existing problems, we propose a novel on-policy grasp detection method, which can train and test on the same distribution with dense pixel-level grasp labels generated on RGB-D images. A Parallel-Depth Grasp Generation (PDG-Generation) method is proposed to generate a parallel depth image through a new imaging model of projecting points in parallel; then this method generates multiple candidate grasps for each pixel and obtains robust grasps through flatness detection, force-closure metric and collision detection. Then, a large comprehensive Pixel-Level Grasp Pose Dataset (PLGP-Dataset) is constructed and released; distinguished with previous datasets with off-policy data and sparse grasp samples, this dataset is the first pixellevel grasp dataset, with the on-policy distribution where grasps are generated based on depth images. Lastly, we build and test a series of pixel-level grasp detection networks with a data augmentation process for imbalance training, which learn grasp poses in a decoupled manner on the input RGB-D images. Extensive experiments show that our on-policy grasp method can largely overcome the gap between simulation and reality, and achieves the state-of-the-art performance. Code and data are provided at https://github.com/liuchunsense/PLGP-Dataset.

show abstract

“…Recent human segmentation methods focus on human instance part segmentation. There are two paradigms for this direction: top-down pipelines [28,60,45,24,59] and bottom-up pipelines [19,27,67,71]. Our framework solves the instance level segmentation with fine-grained attributes recognition, which is much more challenging.…”

Section: Related Workmentioning

confidence: 99%

Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition

Xu¹,

Li²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Human fashion understanding is one important computer vision task since it has the comprehensive information that can be used for real-world applications. In this work, we focus on joint human fashion segmentation and attribute recognition. Contrary to the previous works that separately model each task as a multi-head prediction problem, our insight is to bridge these two tasks with one unified model via vision transformer modeling to benefit each task. In particular, we introduce the object query for segmentation and the attribute query for attribute prediction. Both queries and their corresponding features can be linked via mask prediction. Then we adopt a two-stream query learning framework to learn the decoupled query representations. For attribute stream, we design a novel Multi-Layer Rendering module to explore more fine-grained features. The decoder design shares the same spirits with DETR, thus we name the proposed method Fahsionformer. Extensive experiments on three human fashion datasets including Fashionpedia, ModaNet and Deepfashion illustrate the effectiveness of our approach. In particular, our method with the same backbone achieve relative 10% improvements than previous works in case of a joint metric ( AP mask IoU+F 1 ) for both segmentation and attribute recognition. To the best of our knowledge, we are the first unified end-to-end vision transformer framework for human fashion analysis. We hope this simple yet effective method can serve as a new flexible baseline for fashion analysis. Code will be available at https://github.com/xushilin1/FashionFormer.

show abstract

Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing

Cited by 75 publications

References 58 publications

Deep Hierarchical Semantic Segmentation

Deep Hierarchical Semantic Segmentation

On-Policy Pixel-Level Grasping Across the Gap Between Simulation and Reality

Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition

Contact Info

Product

Resources

About