Extending Linear Dynamical Systems with Dynamic Stream Weights for Audiovisual Speaker Localization

Schymura, Christopher; Isenberg, Tobias; Kolossa, Dorothea

doi:10.1109/iwaenc.2018.8521384

Cited by 4 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This section presents an extension of the state estimation framework proposed in [19] to nonlinear DSs with DSWs and an arbitrary number of independent observations. A generic state estimation algorithm based on the Gaussian filter paradigm is derived and its relation to the standard EKF is discussed.…”

Section: Dynamical System Descriptionmentioning

confidence: 99%

“…In this regard, the framework presented in this paper is most closely related to approaches developed for discrete state spaces based on HMMs in audiovisual ASR [10], [20]- [22]. The primary contribution of the present work is the introduction of DSW-based MDF into continuous DSs with nonlinear dynamics and observations, which is a natural extension of the initial work reported in [19]. Additionally, a novel approach for computing ODSWs incorporating a Dirichlet prior is derived and integrated into a generic learning framework that allows to train DSW prediction models.…”

Section: Introductionmentioning

confidence: 98%

“…This was recently proposed in [19], which presented an inference scheme for continuous state spaces in the regime of linear dynamical systems (DSs) that incorporated dynamic stream weights (DSWs) into the estimation process. DSWs are weighting factors that control the contribution of the individual sensory observations to the estimation process at each timestep.…”

Section: Introductionmentioning

confidence: 99%

“…a regression function or a neural network [10], [20], [22]. This paper introduces an extended framework for MDF using nonlinear DSs based on the method initially proposed in [19]. This method was restricted to a linear Gaussian state space representation and two sensory modalities.…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, a means to obtain ODSWs from fully observed DSs based on a Dirichlet prior imposed on the stream weights is presented. Compared to the previously introduced ODSW estimators using a Gaussian prior [19], [20], this novel approach provides a clear and intuitive probabilistic interpretation. Additionally, a generic learning scheme for DSWs prediction models using application-specific reliability measures is presented.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

Schymura

Kolossa

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

Data fusion plays an important role in many technical applications that require efficient processing of multimodal sensory observations. A prominent example is audiovisual signal processing, which has gained increasing attention in automatic speech recognition, speaker localization and related tasks. If appropriately combined with acoustic information, additional visual cues can help to improve the performance in these applications, especially under adverse acoustic conditions. A dynamic weighting of acoustic and visual streams based on instantaneous sensor reliability measures is an efficient approach to data fusion in this context. This paper presents a framework that extends the well-established theory of nonlinear dynamical systems with the notion of dynamic stream weights for an arbitrary number of sensory observations. It comprises a recursive state estimator based on the Gaussian filtering paradigm, which incorporates dynamic stream weights into a framework closely related to the extended Kalman filter. Additionally, a convex optimization approach to estimate oracle dynamic stream weights in fully observed dynamical systems utilizing a Dirichlet prior is presented. This serves as a basis for a generic parameter learning framework of dynamic stream weight estimators. The proposed system is application-independent and can be easily adapted to specific tasks and requirements. A study using audiovisual speaker tracking tasks is considered as an exemplary application in this work. An improved tracking performance of the dynamic stream weight-based estimation framework over state-of-the-art methods is demonstrated in the experiments.

show abstract

Section: Dynamical System Descriptionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

Schymura

Kolossa

2020

IEEE/ACM Trans. Audio Speech Lang. Process.

Self Cite

View full text Add to dashboard Cite

show abstract

Perception for Humanoid Robots

Roychoudhury,

Khorshidi,

Agrawal

et al. 2023

Curr Robot Rep

View full text Add to dashboard Cite

Purpose of Review The field of humanoid robotics, perception plays a fundamental role in enabling robots to interact seamlessly with humans and their surroundings, leading to improved safety, efficiency, and user experience. This scientific study investigates various perception modalities and techniques employed in humanoid robots, including visual, auditory, and tactile sensing by exploring recent state-of-the-art approaches for perceiving and understanding the internal state, the environment, objects, and human activities. Recent Findings Internal state estimation makes extensive use of Bayesian filtering methods and optimization techniques based on maximum a-posteriori formulation by utilizing proprioceptive sensing. In the area of external environment understanding, with an emphasis on robustness and adaptability to dynamic, unforeseen environmental changes, the new slew of research discussed in this study have focused largely on multi-sensor fusion and machine learning in contrast to the use of hand-crafted, rule-based systems. Human robot interaction methods have established the importance of contextual information representation and memory for understanding human intentions. Summary This review summarizes the recent developments and trends in the field of perception in humanoid robots. Three main areas of application are identified, namely, internal state estimation, external environment estimation, and human robot interaction. The applications of diverse sensor modalities in each of these areas are considered and recent significant works are discussed.

show abstract

Learning Dynamic Stream Weights for Linear Dynamical Systems Using Natural Evolution Strategies

Schymura

Kolossa

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Extending Linear Dynamical Systems with Dynamic Stream Weights for Audiovisual Speaker Localization

Cited by 4 publications

References 20 publications

Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

Perception for Humanoid Robots

Learning Dynamic Stream Weights for Linear Dynamical Systems Using Natural Evolution Strategies

Contact Info

Product

Resources

About