Visual Affordance and Function Understanding

Hassanin, Mohammed; Khan, Salman; Tahtalı, Murat

doi:10.1145/3446370

Cited by 64 publications

(41 citation statements)

References 186 publications

(245 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Part affordances in our dataset are defined in terms of adjectives (e.g., sittable) or nouns (e.g., armrest), which describe how object parts could afford human daily actions and activities. We adopt certain common affordance categories from a comprehensive survey of visual affordance [17], e.g., containment, sittable, support, openable, rollable, display, and wrapgrasp.…”

Section: Affordance Definition and Dataset Constructionmentioning

confidence: 99%

“…Learning and evaluating PartAfford demands collections of 3D objects and their affordance labels for object parts. Prior work on visual affordance learning [17] either focuses on 2D objects and scenes or lacks part-based annotation [6]. Hence, we construct a part-level, cross-category 3D object affordance dataset annotated with 24 affordance categories shared among over 25, 000 3D objects.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PartAfford: Part-level Affordance Discovery from 3D Objects

Chen¹,

Chen²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Section: Affordance Definition and Dataset Constructionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

PartAfford: Part-level Affordance Discovery from 3D Objects

Chen¹,

Chen²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Another line of work explores the use of affordances in RL, robotics, and control, historically through the lens of perception [53], [19], [52]. Affordances have also been discussed previously in reinforcement learning in order to accelerate planning in model-based RL by planning over only a subset of relevant actions [1], [23], [51].…”

Section: Related Workmentioning

confidence: 99%

What Can I Do Here? Learning New Skills by Imagining Visual Affordances

Khazatsky¹,

Nair²,

Jing³

et al. 2021

Preprint

View full text Add to dashboard Cite

A generalist robot equipped with learned skills must be able to perform many tasks in many different environments. However, zero-shot generalization to new settings is not always possible. When the robot encounters a new environment or object, it may need to finetune some of its previously learned skills to accommodate this change. But crucially, previously learned behaviors and models should still be suitable to accelerate this relearning. In this paper, we aim to study how generative models of possible outcomes can allow a robot to learn visual representations of affordances, so that the robot can sample potentially possible outcomes in new situations, and then further train its policy to achieve those outcomes. In effect, prior data is used to learn what kinds of outcomes may be possible, such that when the robot encounters an unfamiliar setting, it can sample potential outcomes from its model, attempt to reach them, and thereby update both its skills and its outcome model. This approach, visuomotor affordance learning (VAL), can be used to train goal-conditioned policies that operate on raw image inputs, and can rapidly learn to manipulate new objects via our proposed affordance-directed exploration scheme. We show that VAL can utilize prior data to solve real-world tasks such drawer opening, grasping, and placing objects in new scenes with only five minutes of online experience in the new scene.

show abstract

“…In more general terms, the way an agent interacts with a particular object is what defines its affordance, and this is strictly related to their relative motion. In the last decades, computer scientists have also being working on this general idea, trying to quantitatively implement it in the fields of computer vision and robotics ( Ardón et al, 2020 ; Hassanin et al, 2021 ). As far as visual affordance is concerned, that is, extracting affordance information from still images and videos, different cognitive tasks have been considered so far, as for example affordance recognition and affordance segmentation, see ( Hassanin et al, 2021 ) for a recent review.…”

Section: Feature Extraction and Conjugate Velocitiesmentioning

confidence: 99%

Visual Features and Their Own Optical Flow

Betti

Boccignone

Faggi

et al. 2021

Front. Artif. Intell.

View full text Add to dashboard Cite

Symmetries, invariances and conservation equations have always been an invaluable guide in Science to model natural phenomena through simple yet effective relations. For instance, in computer vision, translation equivariance is typically a built-in property of neural architectures that are used to solve visual tasks; networks with computational layers implementing such a property are known as Convolutional Neural Networks (CNNs). This kind of mathematical symmetry, as well as many others that have been recently studied, are typically generated by some underlying group of transformations (translations in the case of CNNs, rotations, etc.) and are particularly suitable to process highly structured data such as molecules or chemical compounds which are known to possess those specific symmetries. When dealing with video streams, common built-in equivariances are able to handle only a small fraction of the broad spectrum of transformations encoded in the visual stimulus and, therefore, the corresponding neural architectures have to resort to a huge amount of supervision in order to achieve good generalization capabilities. In the paper we formulate a theory on the development of visual features that is based on the idea that movement itself provides trajectories on which to impose consistency. We introduce the principle of Material Point Invariance which states that each visual feature is invariant with respect to the associated optical flow, so that features and corresponding velocities are an indissoluble pair. Then, we discuss the interaction of features and velocities and show that certain motion invariance traits could be regarded as a generalization of the classical concept of affordance. These analyses of feature-velocity interactions and their invariance properties leads to a visual field theory which expresses the dynamical constraints of motion coherence and might lead to discover the joint evolution of the visual features along with the associated optical flows.

show abstract

Visual Affordance and Function Understanding

Cited by 64 publications

References 186 publications

PartAfford: Part-level Affordance Discovery from 3D Objects

PartAfford: Part-level Affordance Discovery from 3D Objects

What Can I Do Here? Learning New Skills by Imagining Visual Affordances

Visual Features and Their Own Optical Flow

Contact Info

Product

Resources

About