Wearable System for Personalized and Privacy-preserving Egocentric Visual Context Detection using On-device Deep Learning

Abstract

Wearable egocentric visual context detection raises privacy concerns and is rarely personalized or on-device. We created a wearable system, called PAL, with on-device deep learning so that the user images do not have to be sent to the cloud for processing, and can be processed on-device in a real-time, offline, and privacy-preserving manner. PAL enables human-in-the-loop context labeling using wearable audio input/output and a mobile/web application. PAL uses on-device deep learning models for object and face detection, low-shot custom face recognition (~1 training image per person), low-shot custom context recognition (e.g., brushing teeth, ~10 training images per context), and custom context clustering for active learning. We tested PAL with 4 participants, 2 days each, and obtained ~1000 in-the-wild images. The participants found PAL easy-to-use and each model had дt80% accuracy. Thus, PAL supports wearable, personalized, and privacy-preserving egocentric visual context detection using human-in-the-loop, low-shot, and on-device deep learning. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Ubiquitous and mobile computing; Ubiquitous and mobile computing systems and tools.

An Outlook into the Future of Egocentric Vision

“…Steil et al (2019) presented a system which shuts off the video stream when sensitive visual content is detected, only to reactivate it based on the analysis of eye movements recorded by additional eye tracking cameras. Along the same lines, Khan et al (2021) proposed a deep learning based device which detects usercustomised privacy-sensitive content such as objects and faces of specific people in order to serve as a privacy filter, blocking images which do not satisfy the established privacy constraints. Qiu et al (2023) investigated how converting images into rich text descriptions can serve as an effec-tive privacy-preserving approach for passive dietary intake monitoring from egocentric images, as compared to directly storing the input images.…”

PAL: Intelligence Augmentation using Egocentric Visual Context Detection

“…We performed technical evaluations of PAL. We tested PAL's deep learning models with 4 participants for 2 days each (~1000 in-the-wild images) [19]. Each model had over 80% accuracy -Object Detection: 98.8% (F1 = 0.79, ~1000 instances); Face Detection: 88.8% (F1 = 0.9, ~180 instances); Custom Face Recognition: 86.9% (4 faces, 120 instances); Custom Contexts: 87.2% (7 activities, ~350 images); Custom Clusters: 82% (19 contexts, ~300 images).…”

“…PAL also supports user input for human-in-the-loop training of personalized visual contexts. We used on-device models for generic object and face detection, personalized and lowshot custom face and recognition, and semi-supervised custom context clustering [19]. Compared to existing wearable systems, which use at least 100 training images per custom context [21] and do not use privacy-preserving ondevice deep learning, PAL's on-device models for low-shot and continual learning use ~10 training images per context.…”