Neuromorphic Sensory Integration for Combining Sound Source Localization and Collision Avoidance

Abstract

Animals combine various sensory cues with previously acquired knowledge to safely travel towards a target destination. In close analogy to biological systems, we propose a neuromorphic system which decides, based on auditory and visual input, how to reach a sound source without collisions. The development of this sensory integration system, which identifies the shortest possible path, is a key achievement towards autonomous robotics. The proposed neuromorphic system comprises two event based sensors (the eDVS for vision and the NAS for audition) and the SpiNNaker processor. Open loop experiments were performed to evaluate the system performances. In the presence of acoustic stimulation alone, the heading direction points to the direction of the sound source with a Pearson correlation coefficient of 0.89. When visual input is introduced into the network the heading direction always points at the direction of null optical flow closest to the sound source. Hence, the sensory integration network is able to find the shortest path to the sound source while avoiding obstacles. This work shows that a simple, task dependent mapping of sensory information can lead to highly complex and robust decisions.

Live Demonstration: Neuromorphic Sensory Integration for Combining Sound Source Localization and Collision Avoidance

“…This heading direction information controls the position of a pan-tilt unit through a head direction cell network, as proposed in [4]. The system has already been tested in an offline scenario on a wheeled robot [1]. In this context, we demonstrated the ability of the network to select the shorter path around obstacles toward the sound source.…”

“…This live demonstrator comprises a neuromorphic system which points into the direction of a sound source while avoiding obstacles (for detailed information see [1]). The main component of the demonstrator is a spiking neural network (SNN) implemented onto the SpiNNaker board.…”

Event-Based Eccentric Motion Detection Exploiting Time Difference Encoding

“…The sEMD encodes the time-to-travel across the visual field as a number of spikes (where time-to-travel is inversely proportional to velocity). The sEMD's functionality has been evaluated in Brian 2 simulations and on SpiNNaker using real-world data recorded with the Dynamic Vision Sensor (DVS) (Milde et al, 2018;Schoepe et al, 2019). Furthermore, the model has been implemented on a neuromorphic analog CMOS chip and tested successfully.…”

“…We explored the real-world applicability of the underlying motion detection mechanism prior to this work in which we demonstrated the functionality of the underlying given variable contrast and event-rates in natural environments(Milde et al, 2015(Milde et al, , 2018Schoepe et al, 2019).Frontiers in Neuroscience | www.frontiersin.org…”

An Event-Based, Digital Time Difference Encoder Model Implementation for Neuromorphic Systems

“…Milde et al developed an analog Complementary Metal-Oxide-Semiconductor (CMOS) implementation of the TDE, characterized its performances on silicon and applied it to the encoding of Optical Flow (OF). The TDE model has already been used for processing visual [20], auditory and olfactory information. Its universal applicability has high potential for inspiring innovative preprocessing for SNNs, especially supporting close-loop neuromorphic systems with low latency requirements.…”