Brain-inspired multimodal hybrid neural network for robot place recognition

Abstract

Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi–neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.

Gate tunable MoS2 memristive neuron for early fusion multimodal spiking neural network

“…If the number of modalities increases, it can further reduce hardware costs. In the common model, the rst layer of synapses does not share any weights, while the second layer of synapses shares weights among tasks 31,32 .…”

Perceiving like a Bat: Hierarchical 3D Geometric–Semantic Scene Understanding Inspired by a Biomimetic Mechanism

“…As humans, we understand the surrounding environments effortlessly: we receive and transmit high-level instructions and draw up long-distance travel between different cities, and even accurately predict what will happen in the future. When animals are sensing the environment and generating navigation maps, different sensory cues can activate multiple types of sensory cells in their head [ 23 , 24 , 25 ], as illustrated in Figure 1 . Animal neurons can quickly structure spatiotemporal relationships for the surrounding environment [ 26 , 27 , 28 , 29 , 30 , 31 ].…”

A Spatial Location Representation Method Incorporating Boundary Information

“…Physiological studies have indicated that mammals, when freely moving in unfamiliar environments, are capable of maintaining relative spatial relationships to nests or food through specific cognitive mechanisms. This provides them with positional information for navigation in unfamiliar environments and enables real-time updates based on changes in external environmental cues, thus endowing them with strong perceptual abilities in unknown surroundings [1][2][3][4]. However, existing mobile robot technologies fail to utilize distance information between themselves and obstacles or walls to update their current position when facing unexpected obstacles or barriers.…”