Learning Instance-Level N-Ary Semantic Knowledge At Scale For Robots Operating in Everyday Environments

Abstract

Robots operating in everyday environments need to effectively perceive, model, and infer semantic properties of objects. Existing knowledge reasoning frameworks only model binary relations between an object's class label and its semantic properties, unable to collectively reason about object properties detected by different perception algorithms and grounded in diverse sensory modalities. We bridge the gap between multimodal perception and knowledge reasoning by introducing an n-ary representation that models complex, inter-related object properties. To tackle the problem of collecting n-ary semantic knowledge at scale, we propose a transformer neural network that directly generalizes knowledge from observations of object instances. The learned model can reason at different levels of abstraction, effectively predicting unknown properties of objects in different environmental contexts given different amounts of observed information. We quantitatively validate our approach against five prior methods on LINK, a unique dataset we contribute that contains 1457 situated object instances with 15 multimodal properties types and 200 total properties. Compared to the top-performing baseline, a Markov Logic Network, our model obtains a 10% improvement in predicting unknown properties of novel object instances while reducing training and inference time by 150 times. Additionally, we apply our work to a mobile manipulation robot, demonstrating its ability to leverage n-ary reasoning to retrieve objects and actively detect object properties. The code and data are available at https://github.com/wliu88/LINK.

Metarobotics for Industry and Society: Vision, Technologies, and Opportunities

“…This is because symbiotic and thus mutualistic interactions between robots and citizens are envisioned. In this case, inferred knowledge needs to overcome the often handcrafted perspective of citizens [42], [43]. Knowledge is therefore also interpreted from the conceptual perspective of individual robots depending upon, for instance, their operational contexts.…”

“…Knowledge is therefore also interpreted from the conceptual perspective of individual robots depending upon, for instance, their operational contexts. Unsupervised clustering [42] or transformerbased neural networks [43] can be utilized to this end. In combination with human perception and cognition, these skills strengthen reasoning capabilities of decentralized cogDTs.…”

Explainable Knowledge Graph Embedding: Inference Reconciliation for Knowledge Inferences Supporting Robot Actions

“…Prior works that apply KGs to robotics have demonstrated improved robustness in robot behavior by enabling robots to make complex knowledge inferences. Examples include, substituting failed demonstration actions in plans for executable actions [5], finding objects in alternate locations [6], using alternative tools for tasks [1], inferring conditional object properties [7], and interpolating ambiguous end-user commands [2]. Most of the efforts in modeling KGs for robotics have been focused on developing computational frameworks capable of complex knowledge inferences (e.g., learning KG structure for fact prediction).…”