Audio-Visual Voice Activity Detection Using Diffusion Maps

Dov, David; Talmon, Ronen; Cohen, Israel

doi:10.1109/taslp.2015.2405481

Cited by 52 publications

(50 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, due to the challenging problem setting considered in this study, for which the speech is detected at a fine resolution, we found that incorporating the visual information as proposed in [29] does not improve the detection scores. Hence the simulation of [29] is not presented in the plots.…”

Section: Simulation Resultsmentioning

confidence: 80%

“…In addition, we note that we also compared the proposed algorithm to the algorithm we recently presented in [29]. In [29], we proposed a separate representation of each view and estimated voice activity separately based on each representation; then, we fused the views by merging the estimators. However, due to the challenging problem setting considered in this study, for which the speech is detected at a fine resolution, we found that incorporating the visual information as proposed in [29] does not improve the detection scores.…”

Section: Simulation Resultsmentioning

confidence: 99%

“…Before proceeding to the experimental results, we note that the proposed representation and hence the speech indicator are obtained in a batch manner assuming that N consecutive frames are available in advance. Yet, as described in [29], a training set may be used to construct the representation, and then it can be extended to new incoming frames, e.g., using the Nyström method, in an online manner [36].…”

Section: Audio Visual Fusion With Application To Voice Activity Dmentioning

confidence: 99%

See 2 more Smart Citations

Kernel-Based Sensor Fusion With Application to Audio-Visual Voice Activity Detection

Dov

Talmon

Cohen

2016

IEEE Trans. Signal Process.

Self Cite

View full text Add to dashboard Cite

Abstract-In this paper, we address the problem of multiple view data fusion in the presence of noise and interferences. Recent studies have approached this problem using kernel methods, by relying particularly on a product of kernels constructed separately for each view. From a graph theory point of view, we analyze this fusion approach in a discrete setting. More specifically, based on a statistical model for the connectivity between data points, we propose an algorithm for the selection of the kernel bandwidth, a parameter, which, as we show, has important implications on the robustness of this fusion approach to interferences. Then, we consider the fusion of audio-visual speech signals measured by a single microphone and by a video camera pointed to the face of the speaker. Specifically, we address the task of voice activity detection, i.e., the detection of speech and non-speech segments, in the presence of structured interferences such as keyboard taps and office noise. We propose an algorithm for voice activity detection based on the audio-visual signal. Simulation results show that the proposed algorithm outperforms competing fusion and voice activity detection approaches. In addition, we demonstrate that a proper selection of the kernel bandwidth indeed leads to improved performance.

show abstract

Section: Simulation Resultsmentioning

confidence: 80%

Section: Simulation Resultsmentioning

confidence: 99%

Section: Audio Visual Fusion With Application To Voice Activity Dmentioning

confidence: 99%

See 1 more Smart Citation

Kernel-Based Sensor Fusion With Application to Audio-Visual Voice Activity Detection

Dov

Talmon

Cohen

2016

IEEE Trans. Signal Process.

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a result, the DKS can be viewed as a low-pass filter, which controls the spectral bandwidth. In addition, the DKS can be recast in terms of the diffusion distance, a notion of distance induced by diffusion maps [10] that was shown useful in a broad range of applications, e.g., [36,12,27]. For more details see Diffusion maps.…”

Section: Proposed Methodsmentioning

confidence: 99%

Graph of graphs analysis for multiplexed data with application to imaging mass cytometry

Lin

Shnitzer

Talmon

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Hyper spectral imaging, sensor networks, spatial multiplexed proteomics, and spatial transcriptomics assays is a representative subset of distinct technologies from diverse domains of science and engineering that share common data structures. The data in all these modalities consist of high-dimensional multivariate observations (m-dimensional feature space) collected at different spatial positions and therefore can be analyzed using similar computational methodologies. Furthermore, in many studies practitioners collect datasets consisting of multiple spatial assays of this type, each capturing such data from a single biological sample, patient, or hyper spectral image, etc. Each of these spatial assays could be characterized by several regions of interest (ROIs). The focus of this paper is on a particular application, imaging mass cytometry (IMC), which falls into this problem setup. To extract meaningful information from the multi-dimensional observations recorded at different ROIs across different assays, we propose to analyze such datasets using a two-step graph-based approach. We first construct for each ROI a graph representing the interactions between the m covariates and compute an m dimensional vector characterizing the steady state distribution among features. We then use all these mdimensional vectors to construct a graph between the ROIs from all assays. This second graph is subjected to a nonlinear dimension reduction analysis, retrieving the intrinsic geometric representation of the ROIs. Such a representation provides the foundation for efficient and accurate organization of the different ROIs that correlates with their phenotypes. Theoretically, we show that when the ROIs have a particular bi-modal distribution, the new representation gives rise to a better distinction between the two modalities compared to the maximum a posteriori (MAP) estimator. We applied our method to predict the sensitivity to PD-1 axis blockers treatment of lung cancer subjects based on IMC data, achieving 92% accuracy. This serves as empirical evidence that the graph of graphs approach enables us to integrate multiple ROIs and the intrarelationships between the features at each ROI, giving rise to an informative representation that is strongly associated with the phenotypic state of the entire image. Importantly, this approach is applicable to other modalities such as spatial transcriptomics.

show abstract

“…Their goal is to reduce the amount of noise pollution by redesigning the air compressor. The work presented in [28] highlights an improved way to collect audio data for voice activity detection. It uses both audio and visual signals with a supervised learning algorithm to detect what audio frames correspond to human voice.…”

Section: Related Workmentioning

confidence: 99%

A power-efficient audio acquisition system for smart city applications

2020

View full text Add to dashboard Cite

Acoustic noise has adverse effects on human activities. Aside from hearing impairment and stress-related illnesses, it can also interfere with spoken communication, reduce human performance and affect the quality of life. As urbanization is intensifying, the potential benefits of reducing noise pollution in smart-city environments are extensive. Noise levels can be collected and analyzed using a wireless sensor network which can monitor the noise level by using microphones. However, every wireless system struggles in terms of the battery requirements needed for continuous data collection and monitoring. In this paper, the design of a testbed for a smart microphone system is presented. To save power, a microcontroller and an Analog-to-Digital Converter (ADC) dynamically switch between high and low power modes in response to environmental noise. Specifically, the high powered components are triggered by a spike in the acoustic noise level. Three wireless technologies, WiFi (2.4 GHz), Bluetooth Low Energy (BLE) 4.0 and Zigbee were examined. According to the results, the power consumption of a node can be lowered by 97% when idle based on the testbed.

show abstract

Audio-Visual Voice Activity Detection Using Diffusion Maps

Cited by 52 publications

References 41 publications

Kernel-Based Sensor Fusion With Application to Audio-Visual Voice Activity Detection

Kernel-Based Sensor Fusion With Application to Audio-Visual Voice Activity Detection

Graph of graphs analysis for multiplexed data with application to imaging mass cytometry

A power-efficient audio acquisition system for smart city applications

Contact Info

Product

Resources

About