Unsupervised Semantic Segmentation by Distilling Feature Correspondences

Hamilton, Mark F.; Zhang, Zhoutong; Hariharan, Bharath; Snavely, Noah; Freeman, William T.

doi:10.48550/arxiv.2203.08414

Cited by 18 publications

(32 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another solution is to make the representations more explainable, so that the causes of mode-split can be spotted and corrected. We propose that by combining unsupervised semantic segmentation ( Cho et al, 2021 ; Hamilton et al, 2022 ; Xu et al, 2022 ) and bag-of-features (BagNet) ( Brendel and Bethge, 2019 ) training objective, it is possible to produce disentangle features that could map each dimension to pixels. Therefore, whenever mode-split happens and corresponding dimensions are identified, and human interference to either recoding setup or training hyperparameters can be applied.…”

Section: Discussionmentioning

confidence: 99%

Selfee, self-supervised features extraction of animal behaviors

Jia

Guo

et al. 2022

eLife

View full text Add to dashboard Cite

Fast and accurately characterizing animal behaviors is crucial for neuroscience research. Deep learning models are efficiently used in laboratories for behavior analysis. However, it has not been achieved to use an end-to-end unsupervised neural network to extract comprehensive and discriminative features directly from social behavior video frames for annotation and analysis purposes. Here, we report a self-supervised feature extraction (Selfee) convolutional neural network with multiple downstream applications to process video frames of animal behavior in an end-to-end way. Visualization and classification of the extracted features (Meta-representations) validate that Selfee processes animal behaviors in a way similar to human perception. We demonstrate that Meta-representations can be efficiently used to detect anomalous behaviors that are indiscernible to human observation and hint in-depth analysis. Furthermore, time-series analyses of Meta-representations reveal the temporal dynamics of animal behaviors. In conclusion, we present a self-supervised learning approach to extract comprehensive and discriminative features directly from raw video recordings of animal behaviors and demonstrate its potential usage for various downstream applications.

show abstract

Section: Discussionmentioning

confidence: 99%

Selfee, self-supervised features extraction of animal behaviors

Jia

Guo

et al. 2022

eLife

View full text Add to dashboard Cite

show abstract

“…(Siméoni et al, 2021) localizes the objects with a self-supervised transformer. (Hamilton et al, 2022) introduces the feature correspondences that distinguish between different classes. Most recently, a new co-segmentation framework based on DINO feature (Amir et al, 2021) has been proposed and achieves better results on object co-segmentation and part co-segmentation.…”

Section: Related Workmentioning

confidence: 99%

“…Semantic Correspondence across Views Tremendous works have explored and demonstrated the importance of object appearance when generating compact feature correspondence across views (Hénaff et al, 2022;Li et al, 2018). This peculiarity is then utilized in self-supervised 2D semantic segmentation frameworks (Hénaff et al, 2022;Li et al, 2018;Chen et al, 2020) to generate semantic representations by selecting positive and negative pairs with either random or KNN-based rules (Hamilton et al, 2022). Drawing inspiration from these prior arts, we construct the visual feature correspondence for NeRF at the appearance using a heuristic rule.…”

Section: Cross View Appearance Correspondencementioning

confidence: 99%

“…During training, we first render multiple image patches using Equation 1, then we feed them into DINO-ViT to generate feature tensors of shape H × W × C . They are then used to generate the appearance correspondence volume (Teed & Deng, 2020;Hamilton et al, 2022) across views, measuring the similarity between two regions of different views:…”

Section: Cross View Appearance Correspondencementioning

confidence: 99%

“…Distilling Semantic Correspondence into Segmentation Field The correspondence volume F from DINO has been verified it has the potential to effectively recall true label co-occurrence from image collections (Hamilton et al, 2022). We next explore how to learn a segmentation field s by leveraging F .…”

Section: Cross View Appearance Correspondencementioning

confidence: 99%

See 2 more Smart Citations

NeRF-SOS: Any-View Self-supervised Object Segmentation on Complex Scenes

Zhang¹,

Wang²,

Gong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural volumetric representations have shown the potential that Multi-layer Perceptrons (MLPs) can be optimized with multi-view calibrated images to represent scene geometry and appearance without explicit 3D supervision. Object segmentation can enrich many downstream applications based on the learned radiance field. However, introducing hand-crafted segmentation to define regions of interest in a complex real-world scene is non-trivial and expensive as it acquires per view annotation. This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object Segmentation (NeRF-SOS), couples object segmentation and neural radiance field to segment objects in any view within a scene. By proposing a novel collaborative contrastive loss in both appearance and geometry levels, NeRF-SOS encourages NeRF models to distill compact geometry-aware segmentation clusters from their density fields and the self-supervised pre-trained 2D visual features. The self-supervised object segmentation framework can be applied to various NeRF models that both lead to photo-realistic rendering results and convincing segmentation maps for both indoor and outdoor scenarios. Extensive results on the LLFF, BlendedMVS, CO3Dv2, and Tank & Temples datasets validate the effectiveness of NeRF-SOS. It consistently surpasses other 2D-based self-supervised baselines and predicts finer object masks than existing supervised counterparts. Please refer to the video on our project page for more details: https://zhiwenfan.github.io/NeRF-SOS/.Recently, neural volumetric rendering techniques have shown great power in scene reconstruction. Especially, neural radiance field (NeRF) and its variants (Mildenhall et al., 2020a; Barron et al., 2021) adopt multi-layer perceptrons (MLPs) to learn continuous representation and utilize calibrated multi-view images to render unseen views with fine-grained details. Besides rendering quality, the ability of scene understanding has been explored by several recent works (Vora et al., 2021;Yang et al., 2021;Zhi et al., 2021). Nevertheless, they either require dense view annotations to train a heavy 3D backbone for capturing semantic representations (Vora et al., 2021;Yang et al., 2021), or necessitate human intervention to provide sparse semantic labels (Zhi et al., 2021). Recent self-supervised object discovery approaches on neural radiance fields (Yu et al., 2021c;Stelzner et al., 2021) try to decompose objects from givens scenes on the synthetic indoor data. However, still remains a gap to be applied in complex real-world scenarios.

show abstract