From Pixels to Buildings: End-to-end Probabilistic Deep Networks for Large-scale Semantic Mapping

Zheng, Kaiyu; Pronobis, Andrzej

doi:10.1109/iros40897.2019.8967568

Cited by 22 publications

(14 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pronobis and Jensfelt (2012) used a Markov random field to segment a 2D grid map. Zheng et al (2018) inferred the topology of a grid map using a graph-structured sum-product network , whereas Zheng and Pronobis (2019) used a neural network. Armeni et al (2016) focused on a 3D mesh, and proposed a method to parse a building into rooms.…”

Section: Related Workmentioning

confidence: 99%

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Rosinol

Violette

Abate

et al. 2021

The International Journal of Robotics Research

141

View full text Add to dashboard Cite

Humans are able to form a complex mental model of the environment they move in. This mental model captures geometric and semantic aspects of the scene, describes the environment at multiple levels of abstractions (e.g., objects, rooms, buildings), includes static and dynamic entities and their relations (e.g., a person is in a room at a given time). In contrast, current robots’ internal representations still provide a partial and fragmented understanding of the environment, either in the form of a sparse or dense set of geometric primitives (e.g., points, lines, planes, and voxels), or as a collection of objects. This article attempts to reduce the gap between robot and human perception by introducing a novel representation, a 3D dynamic scene graph (DSG), that seamlessly captures metric and semantic aspects of a dynamic environment. A DSG is a layered graph where nodes represent spatial concepts at different levels of abstraction, and edges represent spatiotemporal relations among nodes. Our second contribution is Kimera, the first fully automatic method to build a DSG from visual–inertial data. Kimera includes accurate algorithms for visual–inertial simultaneous localization and mapping (SLAM), metric–semantic 3D reconstruction, object localization, human pose and shape estimation, and scene parsing. Our third contribution is a comprehensive evaluation of Kimera in real-life datasets and photo-realistic simulations, including a newly released dataset, uHumans2, which simulates a collection of crowded indoor and outdoor scenes. Our evaluation shows that Kimera achieves competitive performance in visual–inertial SLAM, estimates an accurate 3D metric–semantic mesh model in real-time, and builds a DSG of a complex indoor environment with tens of objects and humans in minutes. Our final contribution is to showcase how to use a DSG for real-time hierarchical semantic path-planning. The core modules in Kimera have been released open source.

show abstract

Section: Related Workmentioning

confidence: 99%

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Rosinol

Violette

Abate

et al. 2021

The International Journal of Robotics Research

141

View full text Add to dashboard Cite

show abstract

“…These authors later joined both models into an endto-end deep model for semantic mapping in large-scale environments with multiple levels of abstraction, called TopoNets [82], which can perform real-time inference, with novelty detection, for unknown spatial information.…”

Section: Roboticsmentioning

confidence: 99%

Sum-product networks: A survey

Sánchez-Cauce¹,

Paris²,

Vegas³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

A sum-product network (SPN) is a probabilistic model, based on a rooted acyclic directed graph, in which terminal nodes represent probability distributions and non-terminal nodes represent convex sums (weighted averages) and products of probability distributions. They are closely related to probabilistic graphical models, in particular to Bayesian networks with multiple context-specific independencies. Their main advantage is the possibility of building tractable models from data, i.e., models that can perform several inference tasks in time proportional to the number of edges in the graph. They are somewhat similar to neural networks and can address the same kinds of problems, such as image processing and natural language understanding. This paper offers a survey of SPNs, including their definition, the main algorithms for inference and learning from data, several applications, a brief review of software libraries, and a comparison with related models.

show abstract

“…For example, the probabilistic circuit (also called sum-product network) [6], [7], one of our irregular workloads, is used in machine learning for inference and reasoning under uncertainty. It can be used for safety-critical applications like autonomous navigation [4] demanding highly accurate computation, but can also be deployed for simpler applications like human activity classification (running, sitting, etc.) [19], [20], which can tolerate some mispredictions.…”

Section: Precision-scalable Posit Tm Unitmentioning

confidence: 99%

“…In this work, we focus on highly-irregular DAGs resulting from more than 99% sparsity in applications like sparse linear algebra, probabilistic machine learning, robotic localization, drone navigation, etc. [2], [3], [4], [5].…”

Section: Introductionmentioning

confidence: 99%

DPU: DAG Processing Unit for Irregular Graphs with Precision-Scalable Posit Arithmetic in 28nm

Shah,

Olascoaga,

Zhao

et al. 2021

Preprint

View full text Add to dashboard Cite

Computation in several real-world applications like probabilistic machine learning, sparse linear algebra, and robotic navigation, can be modeled as irregular directed acyclic graphs (DAGs). The irregular data dependencies in DAGs pose challenges to parallel execution on general-purpose CPUs and GPUs, resulting in severe under-utilization of the hardware. This paper proposes DPU, a specialized processor designed for the efficient execution of irregular DAGs. The DPU is equipped with parallel compute units that execute different subgraphs of a DAG independently. The compute units can synchronize within a cycle using a hardware-supported synchronization primitive, and communicate via an efficient interconnect to a global banked scratchpad. Furthermore, a precision-scalable posit TM arithmetic unit is developed to enable application-dependent precision. The DPU is taped-out in 28nm CMOS, achieving a speedup of 5.1× and 20.6× over state-of-the-art CPU and GPU implementations on DAGs of sparse linear algebra and probabilistic machine learning workloads. This performance is achieved while operating at a power budget of 0.23W, as opposed to 55W and 98W of the CPU and GPU, resulting in a peak efficiency of 538 GOPS/W with DPU, which is 1350× and 9000× higher than the CPU and GPU, respectively. Thus, with specialized architecture, DPU enables low-power execution of irregular DAG workloads.

show abstract

From Pixels to Buildings: End-to-end Probabilistic Deep Networks for Large-scale Semantic Mapping

Cited by 22 publications

References 18 publications

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Sum-product networks: A survey

DPU: DAG Processing Unit for Irregular Graphs with Precision-Scalable Posit Arithmetic in 28nm

Contact Info

Product

Resources

About