Towards Learning to Perceive and Reason About Liquids

Abstract

Abstract. Recent advances in AI and robotics have claimed many incredible results with deep learning, yet no work to date has applied deep learning to the problem of liquid perception and reasoning. In this paper, we apply fully-convolutional deep neural networks to the tasks of detecting and tracking liquids. We evaluate three models: a single-frame network, multi-frame network, and a LSTM recurrent network. Our results show that the best liquid detection results are achieved when aggregating data over multiple frames and that the LSTM network outperforms the other two in both tasks. This suggests that LSTM-based neural networks have the potential to be a key component for enabling robots to handle liquids using robust, closed-loop controllers.

Reasoning About Liquids via Closed-Loop Simulation

“…In this paper, we utilized a thermal camera to acquire liquid detections to focus the evaluation on our experimental methodology. In the future, we plan to combine our methodology with deep learning methods like the ones in [24,14] to perceive liquids, bypassing the need for a thermal camera. Deep learning can also be applied to perform system identification, i.e., to learn the correct physics models and update them in real-time based on perception.…”

“…However, these works use rather crude liquid simulations for prediction tasks that do not require accurate feedback. Schenck and Fox [24] used a finite element method liquid simulator to train a deep network on the tasks of detecting and tracking liquids. They did not use the simulator to reason about perceived liquid, though.…”

“…Work by Griffith et al [8] used liquids to assist a robot in understanding containers from sensory data. In both [24,25], they use deep networks to both perceive liquids in color images and to reason about their behavior. However, their deep networks are limited to the specific setting they are trained on, and do not have the broad applicability of general liquid simulators.…”

“…", or "How can I move a specific amount of this liquid without spilling?". Toward this goal, our work only assumes that the robot can track 3D mesh models of the objects in its environment and can differentiate between liquid and everything else in its camera observations, both tasks that have been addressed in prior work [26,7,24]. We demonstrate that our closedloop liquid simulation enables a robot to reason about liquids in ways that were infeasible before, such as estimating the amount of water in an opaque container during a pouring task, or detecting partial obstruction in water pipes.…”

Visual closed-loop control for pouring liquids

“…However, for the tasks in this paper, the robot must also be able to detect standing water with no motion, for which optical flow is poorly suited. Instead, we build on our own prior work relating to liquid detection in simulation [7]. We developed a method utilizing fully-convolutional neural networks [19] to label pixels in an image as either liquid or not-liquid.…”