An Adaptable Robot Vision System Performing Manipulation Actions With Flexible Objects

Bodenhagen, Leon; Fugl, Andreas Rune; Jordt, Andreas; Willatzen, Morten; Andersen, Knud A.; Olsen, Martin M.; Koch, Reinhard; Petersen, Henrik Gordon; Krüger, Norbert

doi:10.1109/tase.2014.2320157

Cited by 59 publications

(34 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Left: sparsity patterns of the two vectors are totally different. This can be achieved by solving the optimization problem (9). However, this approach neglects the intrinsic common group structure of the dictionaries.…”

Section: B Joint Kernel Group Sparse Codingmentioning

confidence: 99%

“…Visual object recognition has been extensively studied, and many algorithms and standardized benchmark data sets have been developed [53]. However, the practical manipulation task provides great challenge for vision-based object recognition [9], [43]. For example, it is difficult to distinguish between two bottles of similar appearance, one of which is empty while the other one is full of water.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Visual–Tactile Fusion for Object Recognition

Liu

Sun

et al. 2017

IEEE Trans. Automat. Sci. Eng.

209

View full text Add to dashboard Cite

The camera provides rich visual information regarding objects and becomes one of the most mainstream sensors in the automation community. However, it is often difficult to be applicable when the objects are not visually distinguished. On the other hand, tactile sensors can be used to capture multiple object properties, such as textures, roughness, spatial features, compliance, and friction, and therefore provide another important modality for the perception. Nevertheless, effective combination of the visual and tactile modalities is still a challenging problem. In this paper, we develop a visualtactile fusion framework for object recognition tasks. This paper uses the multivariate-time-series model to represent the tactile sequence and the covariance descriptor to characterize the image. Further, we design a joint group kernel sparse coding (JGKSC) method to tackle the intrinsically weak pairing problem in visual-tactile data samples. Finally, we develop a visual-tactile data set, composed of 18 household objects for validation. The experimental results show that considering both visual and tactile inputs is beneficial and the proposed method indeed provides an effective strategy for fusion. Note to Practitioners-Visual and tactile measurements offer complementary properties that make them particularly suitable for fusion in order to address the robust and accurate recognition of objects, which is a necessity for many automation systems.In this paper, we investigate a widely applicable scenario in grasp manipulation. When identifying an object, the manipulator may see it using the camera and touch it using its hand. Thus, we obtain a pair of test samples, including one image sample and one tactile sample. The manipulator then utilizes this sample pair to identify this object with a classifier that is constructed using the previously collected training samples. However, when collecting training samples, we may collect the image samples and the tactile samples separately. In other words, the training samples may not be paired, while the test samples are paired. This paper addresses this practical problem by developing a JGKSC method, which encourages the effects of the same group, but different atoms. Although our focus is on combining visual and tactile information, the described problem framework is common in the automation community. The algorithm described in this paper can therefore work with weak pairings between a variety of sensors.

show abstract

Section: B Joint Kernel Group Sparse Codingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Visual–Tactile Fusion for Object Recognition

Liu

Sun

et al. 2017

IEEE Trans. Automat. Sci. Eng.

209

View full text Add to dashboard Cite

show abstract

“…Cloth and fabrics have received particular attention [2], [7], [9], [10], [14] due to their natural connections to everyday household life, but the manipulation of flexible materials extends to other fields, including industrial [3], [8] and biomedical [6] applications.…”

Section: Related Workmentioning

confidence: 99%

“…Minimal representations were proposed by Salzmann et al [9], in which the cloth state is represented as a combination of learned deformation modes. However, the standard method for simulating deformable objects is with physics-based models, as in [3], [10].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Collaborative human-robot manipulation of highly deformable materials

Kruse¹,

Radke²,

Wen³

2015

2015 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Robotic manipulation of highly deformable materials is inherently challenging due to the need to maintain tension and the high dimensionality of the state of the material. Past work in this area mostly focuses on generating a detailed model for the material and its interaction with the robot, then using the model to construct a motion plan. In this paper, we take a different approach by using only sensor feedback to dictate the robot motion.We consider the collaborative manipulation of a deformable sheet between a person and a dual-armed robot (Baxter by Rethink Robotics). The robot is capable of contact sensing via joint torque sensors and is equipped with a head-mounted RGBd sensor. The robot senses contact force to maintain tension of the sheet, and in turn comply to the human motion. This is akin to handling a tablecloth with a partner but with one's eyes closed. To improve the response, we use the RGBd sensor to detect folds, and command the robot to move in an orthogonal direction to smooth them out. This is like handling cloth by looking at the cloth itself. Both controllers are able to follow human motion without excessive crimps in the sheet, but as expected, the hybrid controller combining force and vision outperforms the force controller alone in terms of tension force transient. The ability to quickly detect the state of the deformable material also enables more complex manipulation strategies in the future.

show abstract

Multifunctional Optoelectronic Synapse Based on Ferroelectric Van der Waals Heterostructure for Emulating the Entire Human Visual System

Guo

Song

Wong

et al. 2021

Adv Funct Materials

View full text Add to dashboard Cite

The development of optoelectronic synapses can provide an important breakthrough toward creating a sophisticated and adaptable artificial visual system analogous to that of humans. However, it remains a great challenge to implement the various functions of the biological visual neuromorphic system at the single device level. Intriguingly, 2D van der Waals (vdW) heterostructure may offer a platform to address the issue. Here, a novel multifunctional synaptic device based on ferroelectric α-In 2 Se 3 /GaSe vdW heterojunction is proposed to emulate the entire biological visual system. Essential synaptic behaviors are observed in response to light and electrical stimuli; additionally, the retina-like selectivity for light wavelengths and the achievement of Pavlov's dog experiment demonstrate the device's capacity for processing complex electrical and optical inputs. Beyond the optoelectronic synaptic behaviors, the device incorporates memory and logic functions analogous to those in the brain's visual cortex. The results of artificial neural network simulations show that the vdW heterojunction-based device is completely capable of performing logic operations and recognizing images with a high degree of accuracy. The study indicates that versatile devices with a rationally designed construction have great potential for efficiently processing complex visual information and may simplify the design of artificial visual systems.

show abstract

An Adaptable Robot Vision System Performing Manipulation Actions With Flexible Objects

Abstract: Fig. 1: a) Whole setup. b) Grasping process. c) Peg-in-Hole action d) Laying-Down action.

Cited by 59 publications

References 38 publications

Visual–Tactile Fusion for Object Recognition

Visual–Tactile Fusion for Object Recognition

Collaborative human-robot manipulation of highly deformable materials

Multifunctional Optoelectronic Synapse Based on Ferroelectric Van der Waals Heterostructure for Emulating the Entire Human Visual System

Contact Info

Product

Resources

About