Where2Act: From Pixels to Actions for Articulated 3D Objects

Abstract

One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goalwe extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. But more importantly, our learned models even transfer to real-world data. Check the project website for the code and data release.

VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects

“…Learning Actionable Visual Representations aims for learning visual representations that are strongly aware of downstream robotic manipulation tasks and directly indicative of action probabilities for robotic executions, in contrast to predicting standardized visual semantics, such as category labels [48,49], segmentation masks [50,22], and object poses [51,52], which are usually defined independently from any specific robotic manipulation task. Grasping [53,54,55,56,57,58,59,60,61] or manipulation affordance [62,13,63,64,14,65,66,67,15] is one major kind of actionable visual representations, while many other types have been also explored recently (e.g., spatial maps [68,69], keypoints [70,71], contact points [72], etc). Following the recent work Where2Act [15], we employ dense affordance maps as the actionable visual representations to suggest action possibility at every point on 3D articulated objects.…”

“…Grasping [53,54,55,56,57,58,59,60,61] or manipulation affordance [62,13,63,64,14,65,66,67,15] is one major kind of actionable visual representations, while many other types have been also explored recently (e.g., spatial maps [68,69], keypoints [70,71], contact points [72], etc). Following the recent work Where2Act [15], we employ dense affordance maps as the actionable visual representations to suggest action possibility at every point on 3D articulated objects. Extending beyond Where2Act which considers task-less shortterm manipulation, we further augment the per-point action predictions with task-aware distributions of trajectory proposals, providing more actionable information for downstream executions.…”

“…Learning Perception from Interaction augments the tremendously successful learning paradigm using offline curated datasets [48,50,21,22] by allowing learning agents to collect online active data samples, which are more task-aware and learning-efficient, during navigation [73,16], recognition [74,75,76], segmentation [77,78], and manipulation [79,80]. Many works have also demonstrated the usefulness of simulated interactions for learning perception [81,55,82,16,65,83,15] and promising generalizability to the real world [84,85,86,87,88,89,90]. Our method follows the route of learning perception from interaction via using the action trajectories discovered by an RL interaction policy to supervise a jointly trained perception system, which reversely produces curiosity feedback [19] to encourage the RL policy to explore diverse action proposals.…”

“…We propose a new type of actionable visual representations [13,14,15] exploring a more geometryaware, interaction-aware, and task-aware perception-interaction handshaking point for manipulating 3D articulated objects. Concretely, we train the perception system to predict action possibility and visual action trajectory proposals at every point over parts of 3D articulated objects (See Figure 1).…”

“…The proposed actionable visual priors, as a "preparation for future tasks" [16] or "visually-guided plans" [17,18], can provide informative guidance for downstream robotic planning and control. Sharing a similar spirit with [14,15], we formulate our visual action possibility predictions as perpoint affordance maps, on which the downstream robotic planners may sample a position to interact according to the predicted likelihood of success. Then, for a chosen point for interaction, the discrete task planner may search for applicable interaction modes (e.g., whether to attempt a grasp) within a much smaller space formed by the visual action trajectory distribution, instead of searching in the entire solution space.…”