Energy and Computation Efficient Audio-Visual Voice Activity Detection Driven by Event-Cameras

Savran, Arman; Tavarone, Raffaele; Higy, Bertrand; Badino, Leonardo; Bartolozzi, Chiara

doi:10.1109/fg.2018.00055

Cited by 8 publications

(8 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such cases are specifically common in robotics and for some mobile device applications. For instance, as a mobile phone application, we can use our method either for pose-free or pose-dependent extension of voice activity detection which has shown to be power efficient and noise robust [ 6 ]. As the phone stays active for long durations, EC-based processing can save considerable energy, and with the advantage of using the visual channel to cope with noisy acoustic environments.…”

Section: Discussionmentioning

confidence: 99%

“…Recovery of facial motion field and brightness while the camera is in motion were shown in several studies [ 34 , 54 ], where conventional cameras suffer from dynamic range limitations and motion blur. Finally, Savran et al [ 6 ] applied spatio-temporal event-based convolution to locate and detect lip activity, and Li et al [ 7 ] proposed a DNN on audio-visual events for speech recognition.…”

Section: Related Workmentioning

confidence: 99%

“…This technique can be then supported by transfer learning techniques on real datasets. For this reason, we prepared a face pose alignment dataset, which is also included as a subset of an EC-based visual speech dataset [ 6 ]. The acquisition setup is presented in Section 3.1 .…”

Section: Face Pose Alignment Datasetmentioning

confidence: 99%

“…A wide range of facial applications can benefit from our method even though we do not align very precisely for fine facial details. Existing EC-based voice activity detection [ 6 ], visual speech [ 7 ] or face recognition [ 8 ] can be implemented in a robust way against large head pose variations. However, our method does not aim at applications where fine detail alignment is necessary, as in 3D face reconstruction, facial performance capture or face transfer, unless it is used as a bootstrap initialisation technique in hybrid EC/frames systems.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Face Pose Alignment with Event Cameras

Savran

Bartolozzi

2020

Sensors

Self Cite

View full text Add to dashboard Cite

Event camera (EC) emerges as a bio-inspired sensor which can be an alternative or complementary vision modality with the benefits of energy efficiency, high dynamic range, and high temporal resolution coupled with activity dependent sparse sensing. In this study we investigate with ECs the problem of face pose alignment, which is an essential pre-processing stage for facial processing pipelines. EC-based alignment can unlock all these benefits in facial applications, especially where motion and dynamics carry the most relevant information due to the temporal change event sensing. We specifically aim at efficient processing by developing a coarse alignment method to handle large pose variations in facial applications. For this purpose, we have prepared by multiple human annotations a dataset of extreme head rotations with varying motion intensity. We propose a motion detection based alignment approach in order to generate activity dependent pose-events that prevents unnecessary computations in the absence of pose change. The alignment is realized by cascaded regression of extremely randomized trees. Since EC sensors perform temporal differentiation, we characterize the performance of the alignment in terms of different levels of head movement speeds and face localization uncertainty ranges as well as face resolution and predictor complexity. Our method obtained 2.7% alignment failure on average, whereas annotator disagreement was 1%. The promising coarse alignment performance on EC sensor data together with a comprehensive analysis demonstrate the potential of ECs in facial applications.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Face Pose Alignment Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Face Pose Alignment with Event Cameras

Savran

Bartolozzi

2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…The investigation of DNNs together with multimodal spiking sensors is still relatively rare. Previous studies of sensor fusion using DNNs on multimodal spiking sensors include a spiking Deep Belief network with the DVS and DAS [16], hardware equivalents for inference [17], and analog CNNs and RNNs using event-driven spike features [15], Another study used event cameras with audio input on voice recognition [18].…”

Section: Introductionmentioning

confidence: 99%

Lip Reading Deep Network Exploiting Multi-Modal Spiking Visual and Auditory Sensors

Neil

Delbrück

et al. 2019

2019 IEEE International Symposium on Circuits and Systems (ISCAS)

View full text Add to dashboard Cite

This work presents a lip reading deep neural network that fuses the asynchronous spiking outputs of two bio-inspired silicon multimodal sensors: the Dynamic Vision Sensor (DVS) and the Dynamic Audio Sensor (DAS). The fusion network is tested on the GRID visual-audio lipreading dataset. Classification is carried out using event-based features generated from the spikes of the DVS and DAS. Networks are trained separately on the two modalities and also jointly trained on both modalities. The jointly trained network when tested on DVS spike frames alone, showed a relative increase in accuracy of around 23% over that of the single DVS modality network.

show abstract