Low Resource Speech-to-Speech Translation of English videos to Kannada with Lip-Synchronization

Malage, Ritu V.; Ashish, H.; Hukkeri, Sanjana; E, Kavya; Jayashree, R.

doi:10.1109/iciccs56967.2023.10142578

Cited by 3 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Automatic Dubbing, lip movement synchronization refers to aligning and synchronizing the lip movements of a character with the corresponding dubbed audio track [1]. The primary objective of lip movement synchronization is to ensure a seamless lip-syncing effect by closely matching the lip movements of the character with the spoken words [2], [3]. This process involves analyzing the audio waveform and generating precise lip movements that accurately reflect the speech sounds' timing, duration, and intensity [4], [5].…”

Section: A Lip Movement Synchronizationmentioning

confidence: 99%

“…Sophisticated algorithms and machine and deep learning techniques are typically employed in the process of lip movement synchronization in Automatic Dubbing [6]. These algorithms analyze the phonetic or linguistic information in the audio track and map it to predefined or learned visual representations of lip movements [3].…”

Section: A Lip Movement Synchronizationmentioning

confidence: 99%

“…By artificially simulating various asynchronous audiovisual data during training, the AVSE system can better handle the asynchronization problem and improve its speech enhancement performance. Pre-trained convolutional neural network (CNN): The study conducted in 2023 [3] proposes an exhaustive audiovisual system for generating Kannada videos from English videos with accurate lip synchronization, even in dynamic environments and with various faces and voices. The system comprises four stages: speech-to-text, neural machine translation, text-to-speech, and lip synchronization.…”

Section: Attention Mechanism and Neural Machine Translation (Nmt)mentioning

confidence: 99%

“…There is a growing interest in exploring other languages for synchronization. Some studies have investigated languages such as Chinese [13], [23], Spanish [22], Japanese [18], [19], German [4], and Indian [2], [3], [21], aiming to expand the applicability of synchronization techniques to various linguistic contexts.…”

Section: B Rq2: Spoken Languages Used For Synchronizationmentioning

confidence: 99%

“…Limited comparative analysis: Some studies needed more direct comparisons with existing methods or alternative techniques, hindering the assessment of their superiority or limitations [1], [3], [6], [7], [17]- [23], [29], [34], [35].…”

Section: Rq4: Prevalent Limitations In Included Studiesmentioning

confidence: 99%

See 4 more Smart Citations

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

Hassan Alshahrani,

Maashi

2024

IEEE Access

View full text Add to dashboard Cite

This systematic literature review (SLR) explores the topic of Facial Expression and Lip Movement Synchronization of an Audio Track in the context of Automatic Dubbing. This SLR aims to gain insights into the advancements, trends, and limitations of technologies related to this topic. The review protocol was well-defined, and we formulated specific research questions to guide the review process. A comprehensive search strategy included reputable electronic databases and relevant keywords. The study selection process followed inclusion and exclusion criteria, including 32 relevant studies published from 1995 to 2024. We systematically performed data extraction to gather critical information from the selected studies. The findings of this SLR contribute to understanding the current state-of-the-art technologies, identifying research gaps, and informing further advancements in this field.

show abstract

Section: A Lip Movement Synchronizationmentioning

confidence: 99%

Section: A Lip Movement Synchronizationmentioning

confidence: 99%

Section: Attention Mechanism and Neural Machine Translation (Nmt)mentioning

confidence: 99%

Section: B Rq2: Spoken Languages Used For Synchronizationmentioning

confidence: 99%

Section: Rq4: Prevalent Limitations In Included Studiesmentioning

confidence: 99%

See 3 more Smart Citations

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

Hassan Alshahrani,

Maashi

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

A Systematic Survey of Multilingual Speech Transcription and Translation

Vaibhav Ravindra,

Dheeraj K N,

Prof. Vinay Raj

2024

IJARSCT

View full text Add to dashboard Cite

This paper presents an innovative initiative dedicated to revolutionizing multilingual communication by leveraging state-of-the-art technologies such as Artificial Intelligence (AI), NLP, or natural language processing and Machine Learning. With a primary focus on Indian languages, the research aims to develop an advanced system capable of seamlessly transcribing speech across diverse linguistic landscapes. Through the incorporation of cutting-edge algorithms and parallel processing techniques, the proposed system facilitates real-time transcription and translation of multiple languages concurrently. Rigorous experimentation and analysis demonstrate the efficiency of the developed framework in breaking down language barriers and fostering inclusive communication. Furthermore, the paper emphasizes the cultural significance of this technology in promoting global connectivity and celebrating linguistic diversity. Ultimately, this research underscores the transformative potential of technology in facilitating cross-cultural understanding and enabling meaningful interactions within a multilingual society

show abstract

Multilingual Speech Transcription and Translation System

Dheeraj K N,

Vaibhav Ravindra,

Prof. A S Vinay Raj

2024

IJARSCT

View full text Add to dashboard Cite

This project implements a multilingual speech recognition and translation system using Python. The system leverages libraries such as `pygame` for audio playback, `gTTS` for text-to-speech conversion, `speech_recognition` for capturing and recognizing spoken language, and Machine Translation model for translating text between languages. Users interact with the system via a Streamlit-based web interface, where they can select the input and target languages and control the start and stop of the translation process. The program can auto-detect the input language if the user chooses, and it provides real-time feedback by displaying the recognized text, translating it into the chosen target language, and converting the translated text into speech. At the core of the system is a continuous loop that listens for speech input through the microphone, processes the audio to recognize the spoken words, and then translates the recognized text into the desired language. The translated text is not only displayed but also spoken aloud using the text-to-speech functionality. The interface is user-friendly, with clear indications of the current status (listening, processing, recognized text, translated text) and handles errors gracefully, providing feedback in case of issues. This project demonstrates a practical application of speech recognition and translation technologies, offering a seamless and interactive multilingual communication tool

show abstract

Low Resource Speech-to-Speech Translation of English videos to Kannada with Lip-Synchronization

Cited by 3 publications

References 8 publications

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

A Systematic Literature Review: Facial Expression and Lip Movement Synchronization of an Audio Track

A Systematic Survey of Multilingual Speech Transcription and Translation

Multilingual Speech Transcription and Translation System

Contact Info

Product

Resources

About