TitaNet: Neural Model for Speaker Representation with 1D Depth-Wise Separable Convolutions and Global Context

Koluguri, Nithin Rao; Park, Taejin; Ginsburg, Boris

doi:10.1109/icassp43922.2022.9746806

Cited by 45 publications

(20 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, Whisper generates the transcription, while WhisperX [1] corrects and aligns the timestamps to mitigate diarization errors due to time shifts. Following this, MarbleNet [10] executes VAD and segmentation to filter out silences, and TitaNet [12] extracts speaker embeddings to identify speakers for each segment. The results are then associated with timestamps produced by WhisperX, attributing speakers to each word based on timestamps, which are further adjusted using punctuation models to compensate for minor time shifts.The below figure Figure 3 is a pictorial representation of audio extraction as explained in the section.…”

Section: Auditory Memorymentioning

confidence: 99%

Episodic Question Answering for Cognitive Agents

Singh,

Seshadri

2024

Preprint

View full text Add to dashboard Cite

As the demand for artificial intelligence robots and cognitive agents increases, it becomes essential for these agents to comprehend previous encounters and respond to inquiries based on their past experiences. In essence, they need to maintain their memories in an episodic manner. This paper presents a novel approach to address this demand by leveraging the real-life experiences of robots to enrich their knowledge base. To achieve this goal, we employ diverse artificial intelligence techniques, including computer vision, multimodal cross embeddings, speech processing, and generative AI. These methods are utilised to establish a knowledge base that functions as memories for an agent, enabling it to maintain a memory akin to that of a human.To ensure comprehensive memory retention, an agent encounters diverse scenarios such as interacting with individuals, observing conversations, and visiting various locations. To maintain a robust visual and linguistic knowledge base encompassing these experiences, we employ techniques like scene graphs, along with the aforementioned AI methodologies. Existing approaches that involve understanding language and vision used in problem statements such as video question answering, dialogue understanding, or world understanding often overlook the temporal order in which events are observed by the agent or may be restricted to the set of characters or the world in which it has been trained. They struggle to effectively retrieve memories and generate meaningful answers based on this chronological context or in cases where we may rely on a number of past experiences which may reason an event that happened in the future. So, in our study, we've worked on building a solid knowledge base and a way for an agent to remember and link events, just like people do.In conclusion, our work aims to make AI more like humans by helping agents remember and understand events better. This could lead to smarter AI systems that adapt well to different situations in the real world.

show abstract

Section: Auditory Memorymentioning

confidence: 99%

Episodic Question Answering for Cognitive Agents

Singh,

Seshadri

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…For each speaker, four samples were considered as a reference, and only one of the samples was used testing. Then, using the TitaNet [45] pre-trained model, which was trained on 5 datasets (Voxceleb [24] and Voxceleb2 [25], NIST SRE portion of datasets from 2004-2008 (LDC2009E100), Switchboard-Cellular1 and Switchboard-Cellular2 [46], Fisher [47] and Librispeech [48]). We fed x at 1:T as a test and x a0 1:T , ..., x a3 1:T as references to the TitaNet [45] model.…”

Section: Speaker Recognitionmentioning

confidence: 99%

“…Then, using the TitaNet [45] pre-trained model, which was trained on 5 datasets (Voxceleb [24] and Voxceleb2 [25], NIST SRE portion of datasets from 2004-2008 (LDC2009E100), Switchboard-Cellular1 and Switchboard-Cellular2 [46], Fisher [47] and Librispeech [48]). We fed x at 1:T as a test and x a0 1:T , ..., x a3 1:T as references to the TitaNet [45] model. The model produces the 192-dimensional embeddings of each audio sample, for example, v at 1:T .…”

Section: Speaker Recognitionmentioning

confidence: 99%

A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset

Peymanfard¹,

Heydarian²,

Lashini³

et al. 2023

Preprint

View full text Add to dashboard Cite

In recent years, significant progress has been made in automatic lip reading. But these methods require large-scale datasets that do not exist for many low-resource languages. In this paper, we have presented a new multipurpose audio-visual dataset for Persian. This dataset consists of almost 220 hours of videos with 1760 corresponding speakers. In addition to lip reading, the dataset is suitable for automatic speech recognition, audio-visual speech recognition, and speaker recognition. Also, it is the first large-scale lip reading dataset in Persian. A baseline method was provided for each mentioned task. In addition, we have proposed a technique to detect visemes (a visual equivalent of a phoneme) in Persian. The visemes obtained by this method increase the accuracy of the lip reading task by 7% relatively compared to the previously proposed visemes, which can be applied to other languages as well.

show abstract

“…Next, the transcription was diarised using available pretrained models [41,42] with lowest diarisation error rate on our dataset (approx. %1.7) to be able to recognise participants in the session.…”

Section: Transcription and Diarisationmentioning

confidence: 99%

An AI powered system to enhance self‐reflection practice in coaching

Jelodari,

Amirhosseini,

Giraldez‐Hayes

2023

Cognitive Comp and Systems

View full text Add to dashboard Cite

Self‐reflection practice in coaching can help with time management by promoting self‐awareness. Through this process, a coach can identify habits, tendencies and behaviours that may be causing distraction or make them less productive. This insight can be used to make changes in behaviour and establish new habits that promote effective use of time. This can also help the coach to prioritise goals and create a clear roadmap. An AI powered system has been proposed that maps the conversion onto topics and relations that could help the coach with note‐taking and progress identification throughout the session. This system enables the coach to actively self‐reflect on time management and make sure the conversation follows the target framework. This will help the coach to better understand the goal setting, breakthrough moment, and client accountability. The proposed end‐to‐end system is capable of identifying coaching segments (Goal, Option, Reality, and Way forward) across a session with 85% accuracy. Experimental evaluation has also been conducted on the coaching dataset which includes over 1k one‐to‐one English coaching sessions. In regards to the novelty, there are no datasets of such nor study of this kind to enable self‐reflection actively and evaluate in‐session performance of the coach.

show abstract

TitaNet: Neural Model for Speaker Representation with 1D Depth-Wise Separable Convolutions and Global Context

Cited by 45 publications

References 21 publications

Episodic Question Answering for Cognitive Agents

Episodic Question Answering for Cognitive Agents

A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset

An AI powered system to enhance self‐reflection practice in coaching

Contact Info

Product

Resources

About