V2A - Vision to Action: Learning Robotic Arm Actions Based on Vision and Language

Nazarczuk, Michal; Mikolajczyk, Krystian

doi:10.1007/978-3-030-69535-4_44

Cited by 5 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance drop of Mod. [5] and HiTUT [24] is 5.6% and 7.4% lower than the proposed method, SGL. Mod.…”

Section: Outcomes and Analysis For Comparisonmentioning

confidence: 69%

“…2) Connectionist Method: With the significant evolution of connectionist methods in recent years, deep neural networks show impressive strengths of learning semantic and highdimensional features, which improves robustness to various types of input data. Packing everthing into one network, end-to-end learning models [5], [6] are first proposed to directly map natural language and vision to a sequence of low-level actions. The sequence-to-sequence model suffers from the well-known issue of teacher forcing, which leads to the poor performance under test scenarios.…”

Section: A Human Instrution Followingmentioning

confidence: 99%

“…Given that AI2THOR does not support physical modeling of robot-object interaction, Execution evaluation requires the Intersection-of-Union (IoU) of detected and ground-truth masks for objects to be over the 0.5 threshold. Based on [5], Valid Success Rate (VSR) and Invalid Success Rate (ISR) are employed for easy, medium and the first hard, and the second hard scenarios, respectively. VSR evaluates tasks with valid solutions while ISR evaluates ones where there is no valid solution.…”

Section: B Manipulation Metricsmentioning

confidence: 99%

“…{rxu72, hchen657, yunzhi.lin, pvela}@gatech.edu The rise of connectionist approaches provided a means to avoid processing natural language and vision based engineered symbolic representations, by automatically learning visual and linguistic features via deep neural networks. For example, sequence-to-sequence models learn to map raw vision and language input into a sequence of low-level actions [5], [6]. These end-to-end designs suffer from performance drops in testing stage.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following

Xu¹,

Chen²,

Lin³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper investigates robot manipulation based on human instruction with ambiguous requests. The intent is to compensate for imperfect natural language via visual observations. Early symbolic methods, based on manually defined symbols, built modular framework consist of semantic parsing and task planning for producing sequences of actions from natural language requests. Modern connectionist methods employ deep neural networks to automatically learn visual and linguistic features and map to a sequence of low-level actions, in an endto-end fashion. These two approaches are blended to create a hybrid, modular framework: it formulates instruction following as symbolic goal learning via deep neural networks followed by task planning via symbolic planners. Connectionist and symbolic modules are bridged with Planning Domain Definition Language. The vision-and-language learning network predicts its goal representation, which is sent to a planner for producing a task-completing action sequence. For improving the flexibility of natural language, we further incorporate implicit human intents with explicit human instructions. To learn generic features for vision and language, we propose to separately pretrain vision and language encoders on scene graph parsing and semantic textual similarity tasks. Benchmarking evaluates the impacts of different components of, or options for, the vision-and-language learning model and shows the effectiveness of pretraining strategies. Manipulation experiments conducted in the simulator AI2THOR show the robustness of the framework to novel scenarios.

show abstract

“…The performance drop of Mod. [5] and HiTUT [24] is 5.6% and 7.4% lower than the proposed method, SGL. Mod.…”

Section: Outcomes and Analysis For Comparisonmentioning

confidence: 69%

Section: A Human Instrution Followingmentioning

confidence: 99%

Section: B Manipulation Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations