SiamSNN: Siamese Spiking Neural Networks for Energy-Efficient Object Tracking

Luo, Yihao; Xu, Min; Cao, Yuan; Cao, Xiang; Zhang, Liangqi; Xu, Yan; Wang, Tianjiang; Feng, Qianjin

doi:10.1007/978-3-030-86383-8_15

Cited by 18 publications

(8 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this way, events are treated as spikes that can be handled directly by SNN (Jiang et al, 2021 ). SiamSNN (Luo et al, 2021 ), the deep SNN for object tracking, uses the model converted from SiamFC and achieves low precision loss on the benchmarks. But SiamSNN is not directly trained with SNN, it is trained using the conversion algorithm with pretrained ANN.…”

Section: Introductionmentioning

confidence: 99%

SCTN: Event-based object tracking with energy-efficient deep convolutional spiking neural networks

Wang²,

Yan³

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

Event cameras are asynchronous and neuromorphically inspired visual sensors, which have shown great potential in object tracking because they can easily detect moving objects. Since event cameras output discrete events, they are inherently suitable to coordinate with Spiking Neural Network (SNN), which has a unique event-driven computation characteristic and energy-efficient computing. In this paper, we tackle the problem of event-based object tracking by a novel architecture with a discriminatively trained SNN, called the Spiking Convolutional Tracking Network (SCTN). Taking a segment of events as input, SCTN not only better exploits implicit associations among events rather than event-wise processing, but also fully utilizes precise temporal information and maintains the sparse representation in segments instead of frames. To make SCTN more suitable for object tracking, we propose a new loss function that introduces an exponential Intersection over Union (IoU) in the voltage domain. To the best of our knowledge, this is the first tracking network directly trained with SNN. Besides, we present a new event-based tracking dataset, dubbed DVSOT21. In contrast to other competing trackers, experimental results on DVSOT21 demonstrate that our method achieves competitive performance with very low energy consumption compared to ANN based trackers with very low energy consumption compared to ANN based trackers. With lower energy consumption, tracking on neuromorphic hardware will reveal its advantage.

show abstract

Section: Introductionmentioning

confidence: 99%

SCTN: Event-based object tracking with energy-efficient deep convolutional spiking neural networks

Wang²,

Yan³

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

“…where S is the matrix of the size (N Â NÞ containing the sparse coefficients of X. On the other hand, the image can be represented through the measurements Y with fewer samples as follows 29 :…”

Section: Energy Efficient Compressive Sensing Methodsmentioning

confidence: 99%

“…Therefore,

X

is obtained from the measurements with the size of

M \geq italicKlogN

as shown in the following formula 7 :

bold-italicX bold-italicgoodbreak= boldΨ bold-italicS,

where

bold-italicS

is the matrix of the size (

N \times N true)

containing the sparse coefficients of

bold-italicX

. On the other hand, the image can be represented through the measurements

bold-italicY

with fewer samples as follows 29 :

bold-italicY bold-italicgoodbreak= bold-italicϕX bold-italicgoodbreak= bold-italicϕ boldΨ bold-italicS bold-italicgoodbreak+ bold-italicW,

where

bold-italicW

is the additive Gaussian white noise matrix while

bold-italicϕ

is a random measurement matrix of size (

M \times N true)

. Note that if

bold-italicS

K

‐sparse and

M > italicKlogN

, then

bold-italicX

can be reconstructed by solving the

l_{1}

norm minimization problem as follows 32 :

\arg \min {‖bold-italicX‖}_{l_{1}}, s . t . {|(||, bold-italicϕX goodbreak- bold-italicY)|}_{l_{2}} < ϵ,

where

l_{1}

and

l_{2}

are the corresponding norms in Equation ().…”

Section: Energy Efficient Compressive Sensing Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Energy‐efficient compressive sensing for multi‐target tracking in wireless visual sensor networks

Najimi

Sadeghi

2022

Int J Communication

View full text Add to dashboard Cite

Summary Wireless visual sensor networks (WVSN) have vital roles in surveillance applications. In these networks, wireless visual sensors include camera and transceiver module and collect visual information. However, energy consumption and coverage of the tracked targets are important challenges in WVSNs since increasing the coverage leads to increasing energy consumption. Therefore, energy optimization and satisfying the quality of experience (QoE) of the tracked targets are essential issues in these networks. In fact, the appropriate focal length setting leads to increasing the target coverage and quality of the captured targets' images and energy consumption. Therefore, selection of the suitable visual sensors and setting their focal length can overcome the energy consumption challenges and improve QoE of the tracked targets. In this case, compressive sensing is also expected to overcome the battery constraints of the WVSN resources. In this paper, the problem is to minimize the energy consumption of the multi‐target tracking with high reliability, while the coverage and also the quality of the received image of the targets are satisfied by selection of the proper visual sensors and the focal length adjustment. The convex optimization method is used to solve the problem. Also, based on the Karush‐Kuhn‐Tucker conditions, the optimal solution for the problem is obtained. Simulation results validate the efficiency of the proposed method in comparison with the other bench mark algorithms.

show abstract

“…Limited computational resources constrain many application scenarios of downstream vision tasks, and the low-power property of SNNs is well-suited. Currently, SNNs have been applied to several tasks, such as object detection [ 13 , 29 , 50 , 51 , 52 ], optical flow estimation [ 53 , 54 , 55 ], and object tracking [ 56 , 57 ]. Reference [ 58 ] is the first and currently the only SNN work on semantic segmentation.…”

Section: Related Workmentioning

confidence: 99%

Energy-Efficient Spiking Segmenter for Frame and Event-Based Images

2023

View full text Add to dashboard Cite

Semantic segmentation predicts dense pixel-wise semantic labels, which is crucial for autonomous environment perception systems. For applications on mobile devices, current research focuses on energy-efficient segmenters for both frame and event-based cameras. However, there is currently no artificial neural network (ANN) that can perform efficient segmentation on both types of images. This paper introduces spiking neural network (SNN, a bionic model that is energy-efficient when implemented on neuromorphic hardware) and develops a Spiking Context Guided Network (Spiking CGNet) with substantially lower energy consumption and comparable performance for both frame and event-based images. First, this paper proposes a spiking context guided block that can extract local features and context information with spike computations. On this basis, the directly-trained SCGNet-S and SCGNet-L are established for both frame and event-based images. Our method is verified on the frame-based dataset Cityscapes and the event-based dataset DDD17. On the Cityscapes dataset, SCGNet-S achieves comparable results to ANN CGNet with 4.85 × energy efficiency. On the DDD17 dataset, Spiking CGNet outperforms other spiking segmenters by a large margin.

show abstract

SiamSNN: Siamese Spiking Neural Networks for Energy-Efficient Object Tracking

Cited by 18 publications

References 55 publications

SCTN: Event-based object tracking with energy-efficient deep convolutional spiking neural networks

SCTN: Event-based object tracking with energy-efficient deep convolutional spiking neural networks

Energy‐efficient compressive sensing for multi‐target tracking in wireless visual sensor networks

Energy-Efficient Spiking Segmenter for Frame and Event-Based Images

Contact Info

Product

Resources

About