A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing

MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

“…Through extensive experiments, our small model obtains 40.2% and 51.1% of mIoU scores, surpassing the previous multi-task learning baseline [10] by absolute +5.7% and +7.0% on COCOStuff-10K and DMS, respectively. For single-task learning on materials segmentation, i.e., on DMS, our model reaches state-of-the-art performance, having +8.1% mIoU gains compared to previous CNN counterpart [13]. To verify the practicability of our MATERobot for recognizing material categories in real-world scenarios, we conduct a task-oriented user study with six blindfolded participants.…”

“…Recently, Vision Transformer [11], [20] is proposed to utilize the self-attention operation in transformer layers to extract non-local features from a sequence of image patches, yielding an alternative backbone solution compared to convolutional counterparts [19], [18], [21]. In DMS [13], a model based on ResNet [21] is used to address dense material segmentation. In contrast to DMS [13], we adopt Vision Transformer as the backbone, e.g., ViT [11], to perform material semantic segmentation, with the aim of extracting long-distance dependencies between image patches, since the long-range contextual information is crucial for robust representation of diverse materials [10].…”

“…In DMS [13], a model based on ResNet [21] is used to address dense material segmentation. In contrast to DMS [13], we adopt Vision Transformer as the backbone, e.g., ViT [11], to perform material semantic segmentation, with the aim of extracting long-distance dependencies between image patches, since the long-range contextual information is crucial for robust representation of diverse materials [10]. However, due to the high computational demanding of self-attention operation, there is still a bottleneck when deploying a plain vision transformer on resource-constrained mobile platforms, e.g.…”

“…Datasets. We adopt COCOStuff-10K [12] and DMS [13] for general object and material segmentation, respectively. The COCOStuff-10K dataset [12] has 9000/1000 images for training/testing.…”

“…In order to endow MATERobot with robust perception capability, including general and material semantic segmentation, we perform model training on COCOStuff-10K [12] and DMS [13] datasets, both contain more than 10K training samples. Through extensive experiments, our small model obtains 40.2% and 51.1% of mIoU scores, surpassing the previous multi-task learning baseline [10] by absolute +5.7% and +7.0% on COCOStuff-10K and DMS, respectively.…”