Disjoint Mapping Network for Cross-modal Matching of Voices and Faces

Wen, Yandong; Ismail, Mahmoud Al; Liu, Weiyang; Raj, Bhiksha; Singh, Rita

doi:10.48550/arxiv.1807.04836

Cited by 10 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the sample size increases, the accuracy decreases excessively Wen et al [42] . The correlation between modes is utilized Dataset acquisition is difficult Voice-Face Matching Wang et al [55] Can deal with multiple samples Can change the size of input Static image only; model complexity Hoover et al [45] Easy to implement Robust Efficient…”

Section: Methods Ideas and Strengths Weaknessesmentioning

confidence: 99%

“…However, the correlation between the two modalities was not fully utilized in the above method. Therefore, Wen et al [42] proposed a disjoint mapping network (DIMNets) to fully use the covariates (e.g., gender and nationality) [43,44] to bridge the relation between voice and face information. The intuitive assumption was that for a given voice and face pair, the more covariates were shared between the two modalities, the higher the probability of being a match.…”

Section: Voice-facial Matchingmentioning

confidence: 99%

See 1 more Smart Citation

Deep Audio-Visual Learning: A Survey

Zhu

Luo

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities either to improve the performance of previously considered single-modality tasks or to address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods as well as the remaining challenges of each subfield are further discussed. Finally, we summarize the commonly used datasets and performance metrics.

show abstract

Section: Methods Ideas and Strengths Weaknessesmentioning

confidence: 99%

Section: Voice-facial Matchingmentioning

confidence: 99%

Deep Audio-Visual Learning: A Survey

Zhu

Luo

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Kim et al [25] introduce triplet loss to learn overlapping information between faces and voices by using VGG16 [26] and SoundNet [27] for visual and auditory modality respectively. Wen et al [28] propose DIMNet to leverage identity-sensitive factors, such as nationality and gender, as supervision signals to learn a shared representation for different modalities. Based on the strong association between faces and voices, we propose to utilize face embedding to guide models in tracking desirable auditory output.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Target Speech Separation with Voice and Face References

Weber

Wermter

2020

Interspeech 2020

View full text Add to dashboard Cite

Target speech separation refers to isolating target speech from a multi-speaker mixture signal by conditioning on auxiliary information about the target speaker. Different from the mainstream audio-visual approaches which usually require simultaneous visual streams as additional input, e.g. the corresponding lip movement sequences, in our approach we propose the novel use of a single face profile of the target speaker to separate expected clean speech. We exploit the fact that the image of a face contains information about the person's speech sound. Compared to using a simultaneous visual sequence, a face image is easier to obtain by pre-enrollment or on websites, which enables the system to generalize to devices without cameras. To this end, we incorporate face embeddings extracted from a pretrained model for face recognition into the speech separation, which guide the system in predicting a target speaker mask in the time-frequency domain. The experimental results show that a pre-enrolled face image is able to benefit separating expected speech signals. Additionally, face information is complementary to voice reference and we show that further improvement can be achieved when combing both face and voice embeddings 1 .

show abstract

“…The associations between faces and speech have been widely studied in recent years. Cross-modal matching methods by classification [21,22,23,24] and metric learning [25,26] are adopted in identity verification and retrieval. Cross-modal features extracted from faces and speech are applied to disambiguate voiced and unvoiced consonants [27,28]; to track active speakers of a video [29,30]; or to predict emotion [31] and lip motions [28,32] from speech.…”

Section: Related Workmentioning

confidence: 99%

Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging

Bai¹,

Ma²,

Wang³

et al. 2020

Preprint

View full text Add to dashboard Cite

While deep learning technologies are now capable of generating realistic images confusing humans, the research efforts are turning to the synthesis of images for more concrete and application-specific purposes. Facial image generation based on vocal characteristics from speech is one of such important yet challenging tasks. It is the key enabler to influential use cases of image generation, especially for business in public security and entertainment. Existing solutions to the problem of speech2face renders limited image quality and fails to preserve facial similarity due to the lack of quality dataset for training and appropriate integration of vocal features. In this paper, we investigate these key technical challenges and propose Speech Fusion to Face, or SF2F in short, attempting to address the issue of facial image quality and the poor connection between vocal feature domain and modern image generation models. By adopting new strategies on data model and training, we demonstrate dramatic performance boost over state-of-the-art solution, by doubling the recall of individual identity, and lifting the quality score from 15 to 19 based on the mutual information score with VGGFace classifier.

show abstract

Disjoint Mapping Network for Cross-modal Matching of Voices and Faces

Cited by 10 publications

References 19 publications

Deep Audio-Visual Learning: A Survey

Deep Audio-Visual Learning: A Survey

Multimodal Target Speech Separation with Voice and Face References

Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging

Contact Info

Product

Resources

About