Audio Visual Multi-Speaker Tracking with Improved GCF and PMBM Filter

Audiovisual Tracking of Multiple Speakers in Smart Spaces

“…Given the increased availability of easy-to-deploy audio and video sensors and the improvements in computing facilities, in recent years, there has been a relevant growth in the number of proposals for multiple speakers tracking (Multiple-Object Tracking, MOT) in smart spaces, combining audio and video information [ 17 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ]. Qian et al in [ 17 ] conducted an extensive review of state of the art in audiovisual speaker tracking.…”

“…In any case, a clear tendency in the literature proposals is the predominance of probabilistic tracking models such as Bayesian Filters (BFs, i.e., mainly Kalman filters and particle filters) [ 17 , 20 , 29 , 31 , 32 , 34 , 36 , 37 ]. The reasoning behind the use of BFs is that they provide a natural and robust framework to combine different sources of information, in this case, diverse sensing systems.…”

“…Many of these proposed systems track speakers in the image plane (2D) [ 32 , 38 ]. However, proposals addressing the localization in the full 3D space can also be found in [ 17 , 20 , 24 , 27 , 29 , 31 ], thus imposing additional challenges analyzed below.…”

“…The most commonly used approach is based on an observation model that keeps the Bayesian scheme. There are many examples of face detectors based on deep learning [ 25 , 26 , 27 , 28 , 29 , 31 ], although Siamese networks have also been used to generate measures of particle similarity to previous reference images of each target [ 32 , 39 ], and fusion models based on the attention mechanism [ 32 ]. Fewer proposals we found with end-to-end trained audiovisual solutions as in [ 24 , 40 ] for object tracking in which visual and auditory inputs are fused by an added fusion layer.…”

“…In [ 29 ], the authors propose a phase-aware VoiceFilter and a separation before the localization method. They separate the speech from different speakers by first using VoiceFilter with phase estimation and then applying a localization algorithm.…”