Teaching Robots Novel Objects by Pointing at Them

Venkatesh, S.; Upadrashta, Raviteja; Kolathaya, Shishir; Amrutur, Bharadwaj

doi:10.1109/ro-man47096.2020.9223596

Cited by 6 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Eriksen et al [ 26 ] use human supervision to acquire only domain-relevant data, reducing training data requirement. Venkatesh et al [ 27 ] also improves robots object recognition ability with the help of humans. They propose a spatial attention modulation mechanism and teach a robot novel objects it has not encountered before by pointing a hand at the new object of interest.…”

Section: Related Workmentioning

confidence: 99%

A Robot Object Recognition Method Based on Scene Text Reading in Home Environments

Shu-hua

et al. 2021

Sensors

View full text Add to dashboard Cite

With the aim to solve issues of robot object recognition in complex scenes, this paper proposes an object recognition method based on scene text reading. The proposed method simulates human-like behavior and accurately identifies objects with texts through careful reading. First, deep learning models with high accuracy are adopted to detect and recognize text in multi-view. Second, datasets including 102,000 Chinese and English scene text images and their inverse are generated. The F-measure of text detection is improved by 0.4% and the recognition accuracy is improved by 1.26% because the model is trained by these two datasets. Finally, a robot object recognition method is proposed based on the scene text reading. The robot detects and recognizes texts in the image and then stores the recognition results in a text file. When the user gives the robot a fetching instruction, the robot searches for corresponding keywords from the text files and achieves the confidence of multiple objects in the scene image. Then, the object with the maximum confidence is selected as the target. The results show that the robot can accurately distinguish objects with arbitrary shape and category, and it can effectively solve the problem of object recognition in home environments.

show abstract

Section: Related Workmentioning

confidence: 99%

A Robot Object Recognition Method Based on Scene Text Reading in Home Environments

Shu-hua

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Much attention is devoted to object detection in the computer vision literature [20][21] [22][23] [24]. Although end-toend imitation learning does not use object detection, it is also possible to use a pipelined approach where object detection is one module.…”

Section: Related Workmentioning

confidence: 99%

Translating Natural Language Instructions to Computer Programs for Robot Manipulation

Gubbi

Upadrashta

Amrutur

2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

View full text Add to dashboard Cite

It is highly desirable for robots that work alongside humans to be able to understand instructions in natural language. Existing language conditioned imitation learning models directly predict the actuator commands from the image observation and the instruction text. Rather than directly predicting actuator commands, we propose translating the natural language instruction to a Python function which queries the scene by accessing the output of the object detector and controls the robot to perform the specified task. This enables the use of non-differentiable modules such as a constraint solver when computing commands to the robot. Moreover, the labels in this setup are significantly more informative computer programs that capture the intent of the expert rather than teleoperated demonstrations. We show that the proposed method performs better than training a neural network to directly predict the robot actions.

show abstract

“…Much attention is devoted to object detection in the computer vision literature [21][22] [23][24] [25]. Although end-toend imitation learning does not use object detection, it is also possible to use a pipelined approach where object detection is one module.…”

Section: Related Workmentioning

confidence: 99%

Translating Natural Language Instructions to Computer Programs for Robot Manipulation

Venkatesh¹,

Upadrashta²,

Amrutur³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

It is highly desirable for robots that work alongside humans to be able to understand instructions in natural language. Existing language conditioned imitation learning methods predict the actuator commands from the image observation and the instruction text. Rather than directly predicting actuator commands, we propose translating the natural language instruction to a Python function which when executed queries the scene by accessing the output of the object detector and controls the robot to perform the specified task. This enables the use of non-differentiable modules such as a constraint solver when computing commands to the robot. Moreover, the labels in this setup are significantly more descriptive computer programs rather than teleoperated demonstrations. We show that the proposed method performs better than training a neural network to directly predict the robot actions.

show abstract

Teaching Robots Novel Objects by Pointing at Them

Cited by 6 publications

References 16 publications

A Robot Object Recognition Method Based on Scene Text Reading in Home Environments

A Robot Object Recognition Method Based on Scene Text Reading in Home Environments

Translating Natural Language Instructions to Computer Programs for Robot Manipulation

Translating Natural Language Instructions to Computer Programs for Robot Manipulation

Contact Info

Product

Resources

About