Neuromorphic sequence learning with an event camera on routes through vegetation

Abstract

For many robotics applications, it is desirable to have relatively low-power and efficient onboard solutions. We took inspiration from insects, such as ants, that are capable of learning and following routes in complex natural environments using relatively constrained sensory and neural systems. Such capabilities are particularly relevant to applications such as agricultural robotics, where visual navigation through dense vegetation remains a challenging task. In this scenario, a route is likely to have high self-similarity and be subject to changing lighting conditions and motion over uneven terrain, and the effects of wind on leaves increase the variability of the input. We used a bioinspired event camera on a terrestrial robot to collect visual sequences along routes in natural outdoor environments and applied a neural algorithm for spatiotemporal memory that is closely based on a known neural circuit in the insect brain. We show that this method is plausible to support route recognition for visual navigation and more robust than SeqSLAM when evaluated on repeated runs on the same route or routes with small lateral offsets. By encoding memory in a spiking neural network running on a neuromorphic computer, our model can evaluate visual familiarity in real time from event camera footage.

Beyond prediction error: 25 years of modeling the associations formed in the insect mushroom body

“…Some earlier MB-inspired models suggested that such connectivity could allow dynamic recurrent patterns to emerge in KC firing (Payne et al 2010;Arena et al 2013), followed by adaptive linear readout by the MBONs, resembling a reservoir network (Schrauwen et al 2007). A more recent model has explored if adaptive connections between KCs could contribute to learning sequences of sensory patterns in a visual navigation context (Zhu et al 2023).…”

Special Issue—Biosensors and Neuroscience: Is Biosensors Engineering Ready to Embrace Design Principles from Neuroscience?

“…In addition to addressing two distinct grand challenges [25,26], BRAID sought to advance the application of a new class of brain-inspired engineering learning system based on novel or existing neuroscience theories [27][28][29][30][31] in the domain of autonomous systems, including neuromorphic sensors [32][33][34], brain-inspired robots [35][36][37], metacontrollers for multi-agent robots [38][39][40], neuromorphic medical technologies [41,42], and other applications with global economic and health benefits. BRAID especially focused on improving performance metrics beyond energy-and data-efficient learning algorithms [20,43] and learning hardware, including the development of neuromorphic systems [10,[44][45][46], the designs of which were based on insights from recent neuroscience advances [27,31,[47][48][49][50][51][52].…”