Multi-Speaker Tracking From an Audio–Visual Sensing Device

Qian, Xinyuan; Brutti, Alessio; Lanz, Oswald; Omologo, Maurizio; Cavallaro, Andrea

doi:10.1109/tmm.2019.2902489

Cited by 50 publications

(82 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, this approach limits operations to known objects. With previously unseen objects, inference-based approaches localize them in 3D using only partial knowledge, such as dimensions, estimated through uni-modal [20] or multi-modal sensing [21], or by estimating the 6 degrees of freedom (DoF) of textured objects by solving the Perspective-N-Point problem with known 2D-3D correspondences [22], [23]. However, typical objects with textureless, reflective or transparent materials, like those in our benchmark, hinder reliable correspondences [24].…”

Section: B Handoversmentioning

confidence: 99%

Benchmark for Human-to-Robot Handovers of Unseen Containers With Unknown Filling

Sánchez-Matilla

Chatzilygeroudis

Modas

et al. 2020

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

The real-time estimation through vision of the physical properties of objects manipulated by humans is important to inform the control of robots for performing accurate and safe grasps of objects handed over by humans. However, estimating the 3D pose and dimensions of previously unseen objects using only RGB cameras is challenging due to illumination variations, reflective surfaces, transparencies, and occlusions caused both by the human and the robot. In this letter, we present a benchmark for dynamic human-to-robot handovers that do not rely on a motion capture system, markers, or prior knowledge of specific objects. To facilitate comparisons, the benchmark focuses on cups with different levels of transparencies and with an unknown amount of an unknown filling. The performance scores assess the overall system as well as its components in order to help isolate modules of the pipeline that need improvements. In addition to the task description and the performance scores, we also present and distribute as open source a baseline implementation for the overall pipeline to enable comparisons and facilitate progress.

show abstract

Section: B Handoversmentioning

confidence: 99%

Benchmark for Human-to-Robot Handovers of Unseen Containers With Unknown Filling

Sánchez-Matilla

Chatzilygeroudis

Modas

et al. 2020

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The CuNMA is designed to follow the above information. Then, the inter-microphone distance for the closest microphone pairs (1,2), (2,3), (3,4), (4,1), (5,6), (6,7), (7,8), and (8,5) is adjusted as d 1 = 2.3 cm. The inter-microphone distance is d 2 = 3.25 cm for the second subarray with microphone pairs (1,3), (2,4), (5,7), and (6,8).…”

Section: The Proposed Cuboids Nested Microphone Array For Sslmentioning

confidence: 99%

“…Then, the inter-microphone distance for the closest microphone pairs (1,2), (2,3), (3,4), (4,1), (5,6), (6,7), (7,8), and (8,5) is adjusted as d 1 = 2.3 cm. The inter-microphone distance is d 2 = 3.25 cm for the second subarray with microphone pairs (1,3), (2,4), (5,7), and (6,8). This process is repeated for the third subarray with microphone pairs (4,7), (8,3), (5,4), (1,8), (6,1), (5,2), (6,3), and (2,7) with the inter-microphone distance d 3 = 9.2 cm.…”

Section: The Proposed Cuboids Nested Microphone Array For Sslmentioning

confidence: 99%

Section: The Proposed Cuboids Nested Microphone Array For Sslmentioning

confidence: 99%

“…These scenarios highly decrease the accuracy of localization algorithms, which means more error in the estimated locations. Source localization based on the microphone array is one of the principal concepts in array signal processing [1,2], which is implemented in such applications as: automatic camera steering in conferences [3], array steering in speech signal recording [4], robot tracking [5], speech enhancement [6], and speaker tracking [7]. Sound source localization based on microphone array originated from an array of antennas and hydrophones in radar and sonar signal processing [8].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

3D Multiple Sound Source Localization by Proposed Cuboids Nested Microphone Array in Combination with Adaptive Wavelet-Based Subband GEVD

et al. 2020

View full text Add to dashboard Cite

Sound source localization is one of the applicable areas in speech signal processing. The main challenge appears when the aim is a simultaneous multiple sound source localization from overlapped speech signals with an unknown number of speakers. Therefore, a method able to estimate the number of speakers, along with the speaker’s location, and with high accuracy is required in real-time conditions. The spatial aliasing is an undesirable effect of the use of microphone arrays, which decreases the accuracy of localization algorithms in noisy and reverberant conditions. In this article, a cuboids nested microphone array (CuNMA) is first proposed for eliminating the spatial aliasing. The CuNMA is designed to receive the speech signal of all speakers in different directions. In addition, the inter-microphone distance is adjusted for considering enough microphone pairs for each subarray, which prepares appropriate information for 3D sound source localization. Subsequently, a speech spectral estimation method is considered for evaluating the speech spectrum components. The suitable spectrum components are selected and the undesirable components are denied in the localization process. The speech information is different in frequency bands. Therefore, the adaptive wavelet transform is used for subband processing in the proposed algorithm. The generalized eigenvalue decomposition (GEVD) method is implemented in sub-bands on all nested microphone pairs, and the probability density function (PDF) is calculated for estimating the direction of arrival (DOA) in different sub-bands and continuing frames. The proper PDFs are selected by thresholding on the standard deviation (SD) of the estimated DOAs and the rest are eliminated. This process is repeated on time frames to extract the best DOAs. Finally, K-means clustering and silhouette criteria are considered for DOAs classification in order to estimate the number of clusters (speakers) and the related DOAs. All DOAs in each cluster are intersected for estimating the position of the 3D speakers. The closest point to all DOA planes is selected as a speaker position. The proposed method is compared with a hierarchical grid (HiGRID), perpendicular cross-spectra fusion (PCSF), time-frequency wise spatial spectrum clustering (TF-wise SSC), and spectral source model-deep neural network (SSM-DNN) algorithms based on the accuracy and computational complexity of real and simulated data in noisy and reverberant conditions. The results show the superiority of the proposed method in comparison with other previous works.

show abstract