Learning Kinematic Structure Correspondences Using Multi-Order Similarities

Abstract

Abstract-In this paper, we present a novel framework for finding the kinematic structure correspondences between two articulated objects in videos via hypergraph matching. In contrast to appearance and graph alignment based matching methods, which have been applied among two similar static images, the proposed method finds correspondences between two dynamic kinematic structures of heterogeneous objects in videos. Thus our method allows matching the structure of objects which have similar topologies or motions, or a combination of the two. Our main contributions can be summarised as follows: (i) casting the kinematic structure correspondence problem into a hypergraph matching problem by incorporating multi-order similarities with normalising weights, (ii) introducing a structural topology similarity measure by aggregating topology constrained subgraph isomorphisms, (iii) measuring kinematic correlations between pairwise nodes, and (iv) proposing a combinatorial local motion similarity measure using geodesic distance on the Riemannian manifold. We demonstrate the robustness and accuracy of our method through a number of experiments on synthetic and real data, outperforming various other state of the art methods. Our method is not limited to a specific application nor sensor, and can be used as building block in applications such as action recognition, human motion retargeting to robots, and articulated object manipulation amongst others.

Multimodal representation models for prediction and control from partial information

“…We performed extensive evaluation tests of our proposed method. Three different datasets have been used: test data from the robot self-exploration, data from a RGB-D camera of a human playing a piano keyboard, and data from a RGB-D camera of the Imperial-PRL KSC Dataset 3 (data used in [53] to validate kinematic structure correspondences methods). To demonstrate our proposed method in practice, we show that the iCub robot is able to leverage its prediction capability to plan its own actions to imitate a human on the piano keyboard.…”

“…7a). The second test dataset is part of the Imperial-PRL KSC Dataset (data used in [53] to validate kinematic structure correspondences methods). It contains kinect data of a human moving his hands (represented in Fig.…”

Transferring Visuomotor Learning from Simulation to the Real World for Robotics Manipulation Tasks

“…Similarly, Hwang et al [15] investigate the emergence of a mirror neuron system for imitation learning using such a developmental approach. While [15] is limited to imitation learning using the same robot, Chang et al [16], [17] present a method that allows to find body shape correspondence between a number of robots and humans in images. However, this method relies on motion information and is not suited to finding matches of images where the bodies are in the same pose.…”