Learning Object Placements For Relational Instructions by Hallucinating Scene Representations

Mees, Oier; Emek, Alp; Vertens, Johan; Burgard, Wolfram

doi:10.1109/icra40945.2020.9197472

Cited by 14 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Spatial relations also play a crucial role in understanding natural language instructions [4,5], as objects are often described in relation to others in tasks such as object placing [6][7][8] or human robot interaction [2,9,10]. Concretely, spatial relations help the robot disambiguate multiple instances of the same object and to define target areas for placing the picked objects.…”

Section: Related Workmentioning

confidence: 99%

“…Concretely, spatial relations help the robot disambiguate multiple instances of the same object and to define target areas for placing the picked objects. In our previous work, we introduced a novel method to predict pixelwise object placement probability distributions for a set of commonly used prepositions in natural language [7]. In contrast, we relax the assumption of having a single reference object on the tabletop and add a grounding model to effectively place arbitrary objects in a scene that contains multiple objects.…”

Section: Related Workmentioning

confidence: 99%

“…We combine referring expression comprehension with the grounding of spatial relations to enable complex object placement commands such as "place the ball inside the left box". Given an input image I of the scene and the location of the reference item, identified with our aforementioned grounding module, we generate pixelwise object placement probabilities for a set of spatial relations by leveraging the Spatial-RelNet architecture we introduced in our previous work [7]. We consider pairwise relations and express the subject item as being in relation to the reference item.…”

Section: Relational Object Placementmentioning

confidence: 99%

“…3. Our Spatial-RelNet [7] network processes the input RGB image and an object attention mask to produce pixelwise probability maps Γ over a set of spatial relations.…”

Section: Relational Object Placementmentioning

confidence: 99%

“…We make the word embedding of the comprehension and generation modules shared to reduce the number of parameters. For implementation details of Spatial-RelNet, we refer to the original paper [7].…”

Section: Machine Learning Setupmentioning

confidence: 99%

See 4 more Smart Citations

Composing Pick-and-Place Tasks by Grounding Language

Mees

Burgard

2021

Experimental Robotics

Self Cite

View full text Add to dashboard Cite

Controlling robots to perform tasks via natural language is one of the most challenging topics in human-robot interaction. In this work, we present a robot system that follows unconstrained language instructions to pick and place arbitrary objects and effectively resolves ambiguities through dialogues. Our approach infers objects and their relationships from input images and language expressions and can place objects in accordance with the spatial relations expressed by the user. Unlike previous approaches, we consider grounding not only for the picking but also for the placement of everyday objects from language. Specifically, by grounding objects and their spatial relations, we allow specification of complex placement instructions, e.g. "place it behind the middle red bowl". Our results obtained using a real-world PR2 robot demonstrate the effectiveness of our method in understanding pick-and-place language instructions and sequentially composing them to solve tabletop manipulation tasks. Videos are available at http://speechrobot.cs.unifreiburg.de.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Relational Object Placementmentioning

confidence: 99%

“…3. Our Spatial-RelNet [7] network processes the input RGB image and an object attention mask to produce pixelwise probability maps Γ over a set of spatial relations.…”

Section: Relational Object Placementmentioning

confidence: 99%

Section: Machine Learning Setupmentioning

confidence: 99%

See 3 more Smart Citations

Composing Pick-and-Place Tasks by Grounding Language

Mees

Burgard

2021

Experimental Robotics

Self Cite

View full text Add to dashboard Cite

show abstract

Adaptive Prototype Learning and Allocation for Few-Shot Segmentation

Jampani

Sevilla-Lara

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

304

101

View full text Add to dashboard Cite

We introduce One-shot Open Affordance Learning (OOAL), where a model is trained with just one example per base object category, but is expected to identify novel objects and affordances. While vision-language models excel at recognizing novel objects and scenes, they often struggle to understand finer levels of granularity such as affordances. To handle this issue, we conduct a comprehensive analysis of existing foundation models, to explore their inherent understanding of affordances and assess the potential for data-limited affordance learning. We then propose a vision-language framework with simple and effective designs that boost the alignment between visual features and affordance text embeddings. Experiments on two affordance segmentation benchmarks show that the proposed method outperforms state-of-the-art models with less than 1% of the full training data, and exhibits reasonable generalization capability on unseen objects and affordances.

show abstract