Audio-Visual Speaker Localization Using Graphical Models

Kushal, Akash; Rahurkar, Mandar; Li, Feifei; Ponce, Jean; Huang, Tsung‐Wei

doi:10.1109/icpr.2006.284

Cited by 6 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We demonstrated effective use of these paradigms for home care and broad surveillance applications. We are working on extending out current work on using multiple modalities [20] in this framework. Also we are exploring using low-level features for abnormal event detection as shown in Figure 13.…”

Section: Discussionmentioning

confidence: 99%

Event Detection Using "Variable Module Graphs" for Home Care Applications

Sethi

Rahurkar

Huang

2007

EURASIP J. Adv. Signal Process.

View full text Add to dashboard Cite

Technology has reached new heights making sound and video capture devices ubiquitous and affordable. We propose a paradigm to exploit this technology for home care applications especially for surveillance and complex event detection. Complex vision tasks such as event detection in a surveillance video can be divided into subtasks such as human detection, tracking, recognition, and trajectory analysis. The video can be thought of as being composed of various features. These features can be roughly arranged in a hierarchy from low-level features to high-level features. Low-level features include edges and blobs, and high-level features include objects and events. Loosely, the low-level feature extraction is based on signal/image processing techniques, while the high-level feature extraction is based on machine learning techniques. Traditionally, vision systems extract features in a feed-forward manner on the hierarchy, that is, certain modules extract low-level features and other modules make use of these low-level features to extract high-level features. Along with others in the research community, we have worked on this design approach. In this paper, we elaborate on recently introduced V/M graph. We present our work on using this paradigm for developing applications for home care applications. Primary objective is surveillance of location for subject tracking as well as detecting irregular or anomalous behavior. This is done automatically with minimal human involvement, where the system has been trained to raise an alarm when anomalous behavior is detected.

show abstract

Section: Discussionmentioning

confidence: 99%

Event Detection Using "Variable Module Graphs" for Home Care Applications

Sethi

Rahurkar

Huang

2007

EURASIP J. Adv. Signal Process.

View full text Add to dashboard Cite

show abstract

“…speaker). Generative probabilistic models in [2] and [3] for single speaker tracking achieve this by introducing dependencies of both auditory and visual observations on locations in the image plane. Although authors in [2] suggested an enhancement of the model that would tackle the multi-speaker case, it has not been implemented yet.…”

Section: Introductionmentioning

confidence: 99%

Audio-Visual Clustering for 3D Speaker Localization

Khalidov

Forbes

Hansard

et al. 2008

Machine Learning for Multimodal Interaction

View full text Add to dashboard Cite

We address the issue of identifying and localizing individuals in a scene that contains several people engaged in conversation. We use a human-like configuration of sensors (binaural and binocular) to gather both auditory and visual observations. We show that the identification and localization problem can be recast as the task of clustering the audio-visual observations into coherent groups. We propose a probabilistic generative model that captures the relations between audio and visual observations. This model maps the data to a representation of the common 3D scene-space, via a pair of Gaussian mixture models. Inference is performed by a version of the Expectation Maximization algorithm, which provides cooperative estimates of both the activity and the 3D position of each speaker.

show abstract

“…Generative probabilistic models in [2] and [3] for the problem of single speaker tracking achieve this by introducing dependencies of both auditory and visual observations on 2D locations, i.e., in the image plane. Although authors in [2] suggested an enhancement of the model that would tackle the multi-speaker case, it has yet to be implemented.…”

Section: Choosing a Fusion Spacementioning

confidence: 99%

Detection and localization of 3d audio-visual objects using unsupervised clustering

Khalidov

Forbes

Hansard

et al. 2008

Proceedings of the 10th International Conference on Multimodal Interfaces

View full text Add to dashboard Cite

This paper addresses the issues of detecting and localizing objects in a scene that are both seen and heard. We explain the benefits of a human-like configuration of sensors (binaural and binocular) for gathering auditory and visual observations. It is shown that the detection and localization problem can be recast as the task of clustering the audio-visual observations into coherent groups. We propose a probabilistic generative model that captures the relations between audio and visual observations. This model maps the data into a common audio-visual 3D representation via a pair of mixture models. Inference is performed by a version of the expectationmaximization algorithm, which is formally derived, and which provides cooperative estimates of both the auditory activity and the 3D position of each object. We describe several experiments with single-and multiple-speaker detection and localization, in the presence of other audio sources.

show abstract

Audio-Visual Speaker Localization Using Graphical Models

Cited by 6 publications

References 4 publications

Event Detection Using "Variable Module Graphs" for Home Care Applications

Event Detection Using "Variable Module Graphs" for Home Care Applications

Audio-Visual Clustering for 3D Speaker Localization

Detection and localization of 3d audio-visual objects using unsupervised clustering

Contact Info

Product

Resources

About