N-HAR: A Neuromorphic Event-Based Human Activity Recognition System using Memory Surfaces

Pradhan, Bibrat Ranjan; Bethi, Yeshwanth; Narayanan, S. Sankara; Chakraborty, Anirban; Thakur, Chetan Singh

doi:10.1109/iscas.2019.8702581

Cited by 17 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to the relatively new vision-sensing technology, only a few event-based datasets that are captured with an event camera are available. Among these, human pose estimation [2] dataset, action recognition [16]- [18] dataset, face expression recognition [19] dataset, and car recognition [20], [21] datasets. To address the limited availability of event data, researchers have alternatively suggested the generation of semi-synthetic and synthetic event-based datasets.…”

Section: A Event Camera Datasetsmentioning

confidence: 99%

Event Anonymization: Privacy-Preserving Person Re-Identification and Pose Estimation in Event-Based Vision

Ahmad,

Morerio,

Del Bue

2024

IEEE Access

View full text Add to dashboard Cite

The widespread use of visual surveillance in public areas puts individual privacy at stake while also increasing resource usage (energy, bandwidth, and computation). Neuromorphic vision sensors (or event cameras) are considered viable solutions for privacy issues; since event cameras only capture scene dynamics, they do not capture detailed RGB images of individuals. However, recent deep learning architectures have enabled the reconstruction of high-fidelity images from event sensor data that could reveal individual identity information. As a result, it reintroduces privacy risks for event-based vision applications. In this work, we focus on protecting the identity of individuals from such image reconstruction attacks by anonymizing event data. To achieve this, we present an end-to-end network architecture jointly optimized for the twofold objective of preserving privacy and performing a downstream computer vision task. The proposed network learns to scramble events, thereby degrading the quality of images that potential intruders could reconstruct. We demonstrate the application of our framework in two challenging computer vision tasks: person re-identification (ReId) and human pose estimation (HPE). To this end, we also present and evaluate the first event-based person ReId dataset, Event-ReId. We validate the privacy-preserving efficacy of our approach against possible privacy attacks through extensive experiments: for person ReId, we utilize the real event-based Event-ReId dataset and synthetic event data simulated from the SoftBio dataset; for HPE, we use a publicly available event-based dataset DHP19. The results of both tasks show that anonymizing event data effectively protects private information with minimal impact on the subsequent task performance.INDEX TERMS Neuromorphic vision, event camera, event anonymization, privacy-preserving, person re-identification, human pose estimation.

show abstract

Section: A Event Camera Datasetsmentioning

confidence: 99%

Event Anonymization: Privacy-Preserving Person Re-Identification and Pose Estimation in Event-Based Vision

Ahmad,

Morerio,

Del Bue

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Regarding the real-world event-based action recognition dataset, PAF [17] is the first one, which offers 450 recordings spanning 10 categories from an indoor office setting, each with an average length of 5s and a spatial resolution of 346 × 260. N-HAR [19] is another indoor dataset with 3,091 videos, but it is category-unbalanced and contains only 5 actions. DailyAction [14] provides 1,440 recordings across 12 action categories, albeit with a limited spatial resolution of 128 × 128 due to acquisition via DVS128 [77].…”

Section: Datasets For Action Recognitionmentioning

confidence: 99%

“…One of the reasons is the lack of datasets. Although there are existing single-view event action datasets [16], [17], [18], [19], there is a lack of comprehensive multiview event datasets specifically designed for action recognition. DHP19 [18] is the only dataset that can be used for multi-view event-based action recognition, but it is oriented towards pose estimation tasks and is small in scale (33 actions and 2,228 recordings).…”

Section: Introductionmentioning

confidence: 99%

Hypergraph Neural Network for Skeleton-Based Action Recognition

Hao

Guo

et al. 2021

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Action recognition from video data forms a cornerstone with wide-ranging applications. Single-view action recognition faces limitations due to its reliance on a single viewpoint. In contrast, multi-view approaches capture complementary information from various viewpoints for improved accuracy. Recently, event cameras have emerged as innovative bio-inspired sensors, leading to advancements in event-based action recognition. However, existing works predominantly focus on single-view scenarios, leaving a gap in multi-view event data exploitation, particularly in challenges like information deficit and semantic misalignment. To bridge this gap, we introduce HyperMV, a multi-view event-based action recognition framework. HyperMV converts discrete event data into frame-like representations and extracts view-related features using a shared convolutional network. By treating segments as vertices and constructing hyperedges using rule-based and KNN-based strategies, a multi-view hypergraph neural network that captures relationships across viewpoint and temporal features is established. The vertex attention hypergraph propagation is also introduced for enhanced feature fusion. To prompt research in this area, we present the largest multi-view event-based action dataset THU MV-EACT -50, comprising 50 actions from 6 viewpoints, which surpasses existing datasets by over tenfold. Experimental results show that HyperMV significantly outperforms baselines in both cross-subject and cross-view scenarios, and also exceeds the state-of-the-arts in frame-based multi-view action recognition.

show abstract

“…In the past decade, several new kinds of vision sensors have been invented to overcome these disadvantages, and one of them is neuromorphic vision sensors (NVS), which delivers fascinating features such as high temporal resolution, broad dynamic range, and low energy consumption. The characteristics of NVS have attracted increasing attentions from both academia and industry, yielding promising achievements in activity recognition [ 1 ], aided driving [ 2 ], localization [ 3 ], and anomaly detection [ 4 ].…”

Section: Introductionmentioning

confidence: 99%

Event Encryption for Neuromorphic Vision Sensors: Framework, Algorithm, and Evaluation

Wang

et al. 2021

Sensors

View full text Add to dashboard Cite

Nowadays, our lives have benefited from various vision-based applications, such as video surveillance, human identification and aided driving. Unauthorized access to the vision-related data greatly threatens users’ privacy, and many encryption schemes have been proposed to secure images and videos in those conventional scenarios. Neuromorphic vision sensor (NVS) is a brand new kind of bio-inspired sensor that can generate a stream of impulse-like events rather than synchronized image frames, which reduces the sensor’s latency and broadens the applications in surveillance and identification. However, the privacy issue related to NVS remains a significant challenge. For example, some image reconstruction and human identification approaches may expose privacy-related information from NVS events. This work is the first to investigate the privacy of NVS. We firstly analyze the possible security attacks to NVS, including grayscale image reconstruction and privacy-related classification. We then propose a dedicated encryption framework for NVS, which incorporates a 2D chaotic mapping to scramble the positions of events and flip their polarities. In addition, an updating score has been designed for controlling the frequency of execution, which supports efficient encryption on different platforms. Finally, extensive experiments have demonstrated that the proposed encryption framework can effectively protect NVS events against grayscale image reconstruction and human identification, and meanwhile, achieve high efficiency on various platforms including resource-constrained devices.

show abstract

N-HAR: A Neuromorphic Event-Based Human Activity Recognition System using Memory Surfaces

Cited by 17 publications

References 19 publications

Event Anonymization: Privacy-Preserving Person Re-Identification and Pose Estimation in Event-Based Vision

Event Anonymization: Privacy-Preserving Person Re-Identification and Pose Estimation in Event-Based Vision

Hypergraph Neural Network for Skeleton-Based Action Recognition

Event Encryption for Neuromorphic Vision Sensors: Framework, Algorithm, and Evaluation

Contact Info

Product

Resources

About