Unsupervised Segmentation in Real-World Images via Spelke Object Inference

Chen, Honglin; Venkatesh, R.; Friedman, Yoni; Tenenbaum, Joshua B.; Yamins, Daniel; Bear, David M.

doi:10.1007/978-3-031-19818-2_41

Cited by 14 publications

(16 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It will also be important to use additional means of model evaluation, such as model-matched stimuli 35, 61, 75 , stimuli optimized for the model’s predicted response 76, 88, 89 , or directly substituting brain responses into models 90 . And ultimately, analyses such as these need to be related to more fine-grained anatomy and brain response measurements.…”

Section: Discussionmentioning

confidence: 99%

Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

Tuckute

Feather

Boebinger

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep neural networks are commonly used as models of the visual system, but are less explored in audition. Prior work provided examples of audio-trained neural networks that produced good predictions of auditory cortical fMRI responses and exhibited correspondence between model stages and brain regions, but left it unclear whether these results generalize to other neural network models. We evaluated brain-model correspondence for publicly available audio neural network models along with in-house models trained on four different tasks. Most tested models out-predicted previous filter-bank models of auditory cortex, and exhibited systematic model-brain correspondence: middle stages best predicted primary auditory cortex while deep stages best predicted non-primary cortex. However, some state-of-the-art models produced substantially worse brain predictions. The training task influenced the prediction quality for specific cortical tuning properties, with best overall predictions resulting from models trained on multiple tasks. The results suggest the importance of task optimization in constraining brain representations.

show abstract

Section: Discussionmentioning

confidence: 99%

Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

Tuckute

Feather

Boebinger

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, masks recovered from naive image subtraction are imperfect, resulting in worse performance at higher IOU thresholds, which is insufficient for high-performing robotic applications. Optical flow methods can also infer contiguous groups of pixels that move together as the robot is pushing them around [15], [7], [16]. This may be an impractical approach if continuous high-bandwidth video is not available from the robot camera, or if the robot is expected to be grasping objects instead of pushing them in a production environment.…”

Section: B Self-supervised Segmentationmentioning

confidence: 99%

“…An object can be defined as a contiguous group of pixels that move together [7]. When the robot successfully grasps an object, the pixels of that object are removed from the scene.…”

Section: Introductionmentioning

confidence: 99%

Architecture Colorization via Self-supervised Learning and Instance Segmentation

Liu

Chen

2022

Pattern Recognition and Computer Vision

View full text Add to dashboard Cite

Instance segmentation is a fundamental skill for many robotic applications. We propose a self-supervised method that uses grasp interactions to collect segmentation supervision for an instance segmentation model. When a robot grasps an item, the mask of that grasped item can be inferred from the images of the scene before and after the grasp. Leveraging this insight, we learn a grasp segmentation model to segment the grasped object from before and after grasp images. Such a model can segment grasped objects from thousands of grasp interactions without costly human annotation. Using the segmented grasped objects, we can "cut" objects from their original scenes and "paste" them into new scenes to generate instance supervision. We show that our grasp segmentation model provides a 5x error reduction when segmenting grasped objects compared with traditional image subtraction approaches. Combined with our "cut-and-paste" generation method, instance segmentation models trained with our method achieve better performance than a model trained with 10x the amount of labeled data. On a real robotic grasping system, our instance segmentation model reduces the rate of grasp errors by over 3x compared to an image subtraction baseline.

show abstract

“…There's been an explosion of interest in 3D image reconstruction ('A New Trick Lets Artificial Intelligence See in 3D', Wired Magazine [1]), considerable successes in using 3D vision to uncover new biological advances (with DeepMind's AlphaFold [2,3] solving the protein-folding problem), and the suggestion that grounding artificial intelligence in 3D vision will enable better AI (MURI 1 ,(4)(5)(6)). But 3D vision still remains a challenge for AI [7], and is often regarded as the most difficult question facing robotics and autonomous vehicles [8][9][10].At the same time, we are also seeing considerable advances in our understanding of biological vision and navigation. Single-cell recording in freely moving animals has enabled us to understand for the first time how the brain's map of 3D space is organized [11,12], while the emergence of virtual and augmented reality has…”

mentioning

confidence: 99%

“…There's been an explosion of interest in 3D image reconstruction (‘A New Trick Lets Artificial Intelligence See in 3D’, Wired Magazine [1]), considerable successes in using 3D vision to uncover new biological advances (with DeepMind's AlphaFold [2,3] solving the protein-folding problem), and the suggestion that grounding artificial intelligence in 3D vision will enable better AI (MURI 1 , [4–6]). But 3D vision still remains a challenge for AI [7], and is often regarded as the most difficult question facing robotics and autonomous vehicles [8–10].…”

mentioning

confidence: 99%

New Approaches to 3D Vision

Linton

Morgan

Read

et al. 2022

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

New approaches to 3D vision are enabling new advances in artificial intelligence and autonomous vehicles, a better understanding of how animals navigate the 3D world, and new insights into human perception in virtual and augmented reality. Whilst traditional approaches to 3D vision in computer vision (SLAM: simultaneous localization and mapping), animal navigation (cognitive maps), and human vision (optimal cue integration) start from the assumption that the aim of 3D vision is to provide an accurate 3D model of the world, the new approaches to 3D vision explored in this issue challenge this assumption. Instead, they investigate the possibility that computer vision, animal navigation, and human vision can rely on partial or distorted models or no model at all. This issue also highlights the implications for artificial intelligence, autonomous vehicles, human perception in virtual and augmented reality, and the treatment of visual disorders, all of which are explored by individual articles. This article is part of a discussion meeting issue ‘New approaches to 3D vision’.

show abstract

Unsupervised Segmentation in Real-World Images via Spelke Object Inference

Cited by 14 publications

References 26 publications

Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

Architecture Colorization via Self-supervised Learning and Instance Segmentation

New Approaches to 3D Vision

Contact Info

Product

Resources

About