Siamese Capsule Network for End-to-End Speaker Recognition in the Wild

Hajavi, Amirhossein; Etemad, Ali

doi:10.1109/icassp39728.2021.9414722

Cited by 15 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their uses include granting access to individuals [1] who intend to use the products/services provided by the smart devices or customize the provided services by personalizing the experience towards each user [2]. Recently, deep neural networks (DNN) have become the predominant mechanism used in SR systems [3,4,5,6,7,8]. This is mainly due to factors such as improved performance in comparison to traditional SR techniques as a result of state-of-the-art neural architectures and loss functions used in training [9].…”

Section: Introductionmentioning

confidence: 99%

A Deep Neural Network for Short-Segment Speaker Recognition

Hajavi¹,

Etemad²

2019

Interspeech 2019

View full text Add to dashboard Cite

Today's interactive devices such as smart-phone assistants and smart speakers often deal with short-duration speech segments. As a result, speaker recognition systems integrated into such devices will be much better suited with models capable of performing the recognition task with short-duration utterances. In this paper, a new deep neural network, UtterIdNet, capable of performing speaker recognition with short speech segments is proposed. Our proposed model utilizes a novel architecture that makes it suitable for short-segment speaker recognition through an efficiently increased use of information in short speech segments. UtterIdNet has been trained and tested on the VoxCeleb datasets, the latest benchmarks in speaker recognition. Evaluations for different segment durations show consistent and stable performance for short segments, with significant improvement over the previous models for segments of 2 seconds, 1 second, and especially sub-second durations (250 ms and 500 ms).

show abstract

Section: Introductionmentioning

confidence: 99%

A Deep Neural Network for Short-Segment Speaker Recognition

Hajavi¹,

Etemad²

2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…Other research works on in-the-wild scenarios include those of Hajavi &and Etemad [ 23 ] on speaker-recognition tasks and Nguyen et al [ 24 ] for animal-recognition tasks. Hajavi and Etemad 2021 [ 23 ] proposed a Siamese network architecture using capsules and dynamic routing for speaker verification in the wild. Their experimental results on the VoxCeleb dataset gave an error rate (EER) of 3.14%.…”

Section: Related Work On Multimodal-sensor Architectures For Speech R...mentioning

confidence: 99%

Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild

Seng

Ang

2023

Sensors

View full text Add to dashboard Cite

This paper investigates multimodal sensor architectures with deep learning for audio-visual speech recognition, focusing on in-the-wild scenarios. The term “in the wild” is used to describe AVSR for unconstrained natural-language audio streams and video-stream modalities. Audio-visual speech recognition (AVSR) is a speech-recognition task that leverages both an audio input of a human voice and an aligned visual input of lip motions. However, since in-the-wild scenarios can include more noise, AVSR’s performance is affected. Here, we propose new improvements for AVSR models by incorporating data-augmentation techniques to generate more data samples for building the classification models. For the data-augmentation techniques, we utilized a combination of conventional approaches (e.g., flips and rotations), as well as newer approaches, such as generative adversarial networks (GANs). To validate the approaches, we used augmented data from well-known datasets (LRS2—Lip Reading Sentences 2 and LRS3) in the training process and testing was performed using the original data. The study and experimental results indicated that the proposed AVSR model and framework, combined with the augmentation approach, enhanced the performance of the AVSR framework in the wild for noisy datasets. Furthermore, in this study, we discuss the domains of automatic speech recognition (ASR) architectures and audio-visual speech recognition (AVSR) architectures and give a concise summary of the AVSR models that have been proposed.

show abstract

“…D EEP audio representation learning has recently attracted significant interest, specially in applications such as speaker recognition (SR) [9], [10], [43], [31], [11] and speech emotion recognition (SER) [1], [18], [25]. The goal in deep audio representation learning is to learn embeddings from audio or visual signals, which could be used in retrieving…”

Section: Introductionmentioning

confidence: 99%

“…information such as identity or the emotional state of the speaker. This goal is generally best achieved when multimodal audio-visual inputs are used [31], [19], [2] as opposed to when only a single modality of audio or video is used [9], [10], [43], [23], [24], [11]. Nonetheless, in many realworld scenarios, both modalities may not be simultaneously available at inference, resulting in the inability of the model to perform effectively.…”

Section: Introductionmentioning

confidence: 99%

Audio Representation Learning by Distilling Video as Privileged Information

Hajavi

Etemad

2024

IEEE Trans. Artif. Intell.

View full text Add to dashboard Cite

Deep audio representation learning using multimodal audio-visual data often leads to a better performance compared to uni-modal approaches. However, in real-world scenarios both modalities are not always available at the time of inference, leading to performance degradation by models trained for multi-modal inference. In this work, we propose a novel approach for deep audio representation learning using audiovisual data when the video modality is absent at inference. For this purpose, we adopt teacher-student knowledge distillation under the framework of learning using privileged information (LUPI). While the previous methods proposed for LUPI use softlabels generated by the teacher, in our proposed method we use embeddings learned by the teacher to train the student network. We integrate our method in two different settings: sequential data where the features are divided into multiple segments throughout time, and non-sequential data where the entire features are treated as one whole segment. In the non-sequential setting both the teacher and student networks are comprised of an encoder component and a task header. We use the embeddings produced by the encoder component of the teacher to train the encoder of the student, while the task header of the student is trained using ground-truth labels. In the sequential setting, the networks have an additional aggregation component that is placed between the encoder and task header. We use two sets of embeddings produced by the encoder and aggregation component of the teacher to train the student. Similar to the non-sequential setting, the task header of the student network is trained using groundtruth labels. We test our framework on two different audiovisual tasks, namely speaker recognition and speech emotion recognition. Through these experiments we show that by treating the video modality as privileged information for the main goal of audio representation learning, our method results in considerable improvements over sole audio-based recognition as well as prior works that use LUPI.

show abstract

Siamese Capsule Network for End-to-End Speaker Recognition in the Wild

Cited by 15 publications

References 28 publications

A Deep Neural Network for Short-Segment Speaker Recognition

A Deep Neural Network for Short-Segment Speaker Recognition

Multimodal Sensor-Input Architecture with Deep Learning for Audio-Visual Speech Recognition in Wild

Audio Representation Learning by Distilling Video as Privileged Information

Contact Info

Product

Resources

About