Representing Spatial Object Relations as Parametric Polar Distribution for Scene Manipulation Based on Verbal Commands

Kartmann, Rainer; Zhou, You; Liu, Danqing; Paus, Fabian; Asfour, Tamim

doi:10.1109/iros45743.2020.9340925

Cited by 8 publications

(6 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural representations have emerged predicting pixel positions (Venkatesh et al 2021) or pixel-wise probabilities (Mees et al 2020) for placement tasks. To overcome limitations associated with pixel-based distributions, researchers have used parametric probability distributions, such as a polar distribution (Kartmann et al 2020), a mixture of Gaussian distributions (Zhao, Lee, and Hsu 2023), and a Boltzmann distribution (Gkanatsios et al 2023). Our proposed method adopts the polar distribution as a basis for modeling spatial concepts, avoiding the need to predefine the number of components as required by Gaussian mixture models (Kartmann et al 2020;Paxton et al 2022).…”

Section: Related Workmentioning

confidence: 99%

“…To overcome limitations associated with pixel-based distributions, researchers have used parametric probability distributions, such as a polar distribution (Kartmann et al 2020), a mixture of Gaussian distributions (Zhao, Lee, and Hsu 2023), and a Boltzmann distribution (Gkanatsios et al 2023). Our proposed method adopts the polar distribution as a basis for modeling spatial concepts, avoiding the need to predefine the number of components as required by Gaussian mixture models (Kartmann et al 2020;Paxton et al 2022). Further, our method considers the order of expressions and the semantic and geometric relations among objects, allowing for handling semantically identical objects.…”

Section: Related Workmentioning

confidence: 99%

“…We represent the probability distribution as a mixture of instance-wise polar distributions, where a polar distribution is a joint probability density function of two random variables, distance d ∈ R ≥0 and angle ϕ ∈ [−π, π], in the polar coordinate system. As Kartmann et al (2020), we assume that d and ϕ follow a Gaussian distribution N and a Von Mises (i.e., circular) distribution M, respectively;…”

Section: Spatial Distributionmentioning

confidence: 99%

See 2 more Smart Citations

LINGO-Space: Language-Conditioned Incremental Grounding for Space

Kim,

Oh,

Hwang

et al. 2024

AAAI

View full text Add to dashboard Cite

We aim to solve the problem of spatially localizing composite instructions referring to space: space grounding. Compared to current instance grounding, space grounding is challenging due to the ill-posedness of identifying locations referred to by discrete expressions and the compositional ambiguity of referring expressions. Therefore, we propose a novel probabilistic space-grounding methodology (LINGO-Space) that accurately identifies a probabilistic distribution of space being referred to and incrementally updates it, given subsequent referring expressions leveraging configurable polar distributions. Our evaluations show that the estimation using polar distributions enables a robot to ground locations successfully through 20 table-top manipulation benchmark tests. We also show that updating the distribution helps the grounding method accurately narrow the referring space. We finally demonstrate the robustness of the space grounding with simulated manipulation and real quadruped robot navigation tasks. Code and videos are available at https://lingo-space.github.io.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Spatial Distributionmentioning

confidence: 99%

See 1 more Smart Citation

LINGO-Space: Language-Conditioned Incremental Grounding for Space

Kim,

Oh,

Hwang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Bias can be introduced by choosing a model whose structure matches that of the problem at hand. For the problem of placing objects according to desired spatial relations, we proposed to represent spatial relations as parametric probability distributions defined in cylindrical coordinates in our previous work ( Kartmann et al., 2020 ; 2021 ), observing that capturing relative positions in terms of horizontal distance, horizontal direction, and vertical distance closely matches the notions of common spatial prepositions.…”

Section: Introductionmentioning

confidence: 99%

Interactive and incremental learning of spatial object relations from human demonstrations

Kartmann

Asfour

2023

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Humans use semantic concepts such as spatial relations between objects to describe scenes and communicate tasks such as “Put the tea to the right of the cup” or “Move the plate between the fork and the spoon.” Just as children, assistive robots must be able to learn the sub-symbolic meaning of such concepts from human demonstrations and instructions. We address the problem of incrementally learning geometric models of spatial relations from few demonstrations collected online during interaction with a human. Such models enable a robot to manipulate objects in order to fulfill desired spatial relations specified by verbal instructions. At the start, we assume the robot has no geometric model of spatial relations. Given a task as above, the robot requests the user to demonstrate the task once in order to create a model from a single demonstration, leveraging cylindrical probability distribution as generative representation of spatial relations. We show how this model can be updated incrementally with each new demonstration without access to past examples in a sample-efficient way using incremental maximum likelihood estimation, and demonstrate the approach on a real humanoid robot.

show abstract

“…Previously, researchers have worked on modeling, interpreting, and grounding relations between scene elements [5], [6], [7], [8]. Such reasoning can enable robots to move a query object with respect to an anchor object to satisfy a desired spatial relations [9], [10], [3], [11], [12], [13]. More complex spatial structures from language instructions have be generated in a blocks world environment by chaining these binary relation changing manipulations [14].…”

Section: Introductionmentioning

confidence: 99%

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Liu¹,

Paxton²,

Hermans³

et al. 2021

Preprint

View full text Add to dashboard Cite

Rearrange objects that are smaller than the green glass pan tower, top, left, west line, top, left, large circle, top, right, large, north Rearrange objects that have the same color as the glass stapler tower, top, right, west line, bottom, left, large circle, top, left, small, west Rearrange yellow objects circle, bottom, right, medium circle, top, middle, medium circle, top, middle, large Rearrange yellow objects Rearrange objects that have the same material as the blue object circle, bottom, right, large circle, bottom, middle, medium circle, bottom, middle, large Set the table Rearrange objects that are smaller than the green glass pan tower, top, left, west line, top, left, large circle, top, right, large, north Rearrange objects that have the same color as the glass stapler tower, top, right, west line, bottom, left, large circle, top, left, small, west Rearrange yellow objects circle, bottom, right, medium circle, top, middle, medium circle, top, middle, large Rearrange yellow objects Rearrange objects that have the same material as the blue object circle, bottom, right, large circle, bottom, middle, medium circle, bottom, middle, large Set the table Put objects in a line Set the table Put objects in a line Make a circle

show abstract

Representing Spatial Object Relations as Parametric Polar Distribution for Scene Manipulation Based on Verbal Commands

Cited by 8 publications

References 28 publications

LINGO-Space: Language-Conditioned Incremental Grounding for Space

LINGO-Space: Language-Conditioned Incremental Grounding for Space

Interactive and incremental learning of spatial object relations from human demonstrations

StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects

Contact Info

Product

Resources

About