Interpreting Character Embeddings With Perceptual Representations: The Case of Shape, Sound, and Color

Boldsen, Sidsel; Agirrezabal, Manex; Hollenstein, Nora

doi:10.18653/v1/2022.acl-long.470

Cited by 2 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the coefficient drops through the training of the weight matrix in the baseline and the Proposed (Trainable) methods, this comparison points out that the embeddings learned from the phoneme co-occurrences in sentences do not clearly distinguish consonants and vowels. This contradicts the fact that such neural embeddings are known to represent phonetic relationships quite well [2,13]. Yet, since even the baseline performs moderately in the mAP and rank correlation metrics, we can also confirm that the neural embeddings can learn relationships within consonants and within vowels even without any explicit prior.…”

Section: Resultsmentioning

confidence: 66%

“…Recent Natural Language Processing (NLP) techniques also obtain neural phoneme embeddings that reflect phonetic similarity without explicit prior and supervision [2,13]. Kolachina and Magyar [13] evaluate if Word2vec [15,16] can learn the phonetic relationships among phonemes.…”

Section: Computational Approaches To Phoneticsmentioning

confidence: 99%

“…They feed the model with corpora of artificial languages as well as English, concluding that Word2vec embeddings capture the phonetic relationships from such corpora quite well. Boldsen et al [2] perform a similar analysis of character embeddings in multiple languages using Long Short-Term Memory (LSTM) and Transformer by training these models for the next character prediction task. The obtained character embeddings show strong correlations to the sound representations derived from phoneme classifications in English, which supports the results of the previous work [13].…”

Section: Computational Approaches To Phoneticsmentioning

confidence: 99%

See 2 more Smart Citations

IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining

Matsuhira¹,

Kastner²,

Komamizu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recently, large-scale Vision and Language (V&L) pretraining has become the standard backbone of many multimedia systems. While it has shown remarkable performance even in unseen situations, it often performs in ways not intuitive to humans. Particularly, they usually do not consider the pronunciation of the input, which humans would utilize to understand language, especially when it comes to unknown words. Thus, this paper inserts phonetic prior into Contrastive Language-Image Pretraining (CLIP), one of the V&L pretrained models, to make it consider the pronunciation similarity among its pronunciation inputs. To achieve this, we first propose a phoneme embedding that utilizes the phoneme relationships provided by the International Phonetic Alphabet (IPA) chart as a phonetic prior. Next, by distilling the frozen CLIP text encoder, we train a pronunciation encoder employing the IPA-based embedding. The proposed model named IPA-CLIP comprises this pronunciation encoder and the original CLIP encoders (image and text). Quantitative evaluation reveals that the phoneme distribution on the embedding space represents phonetic relationships more accurately when using the proposed phoneme embedding. Furthermore, in some multimodal retrieval tasks, we confirm that the proposed pronunciation encoder enhances the performance of the text encoder and that the pronunciation encoder handles nonsense words in a more phonetic manner than the text encoder. Finally, qualitative evaluation verifies the correlation between the pronunciation encoder and human perception regarding pronunciation similarity.

show abstract

Section: Resultsmentioning

confidence: 66%

Section: Computational Approaches To Phoneticsmentioning

confidence: 99%

Section: Computational Approaches To Phoneticsmentioning

confidence: 99%

See 1 more Smart Citation

IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining

Matsuhira¹,

Kastner²,

Komamizu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This implementation makes the baseline models inaccessible to the phonetic relationship among phonemes, while the proposed models can. However, it should also be noted that character/phoneme-level language models can learn such relationships to some extent implicitly from the phonological restrictions of a language compiled in the training data [36]- [38]. For instance, in every language, since consonants and vowels usually occur in different contexts, language models can implicitly learn which characters/phonemes represent consonants or vowels.…”

Section: ) Baseline and Model Settingsmentioning

confidence: 99%

Interpolating the Text-to-Image Correspondence Based on Phonetic and Phonological Similarities for Nonword-to-Image Generation

Matsuhira,

Kastner,

Komamizu

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Text-to-Image (T2I) generation is the task of synthesizing images corresponding to a given text input. The recent innovations in artificial intelligence have enhanced the capacity of conventional T2I generation, yielding more and more powerful models day by day. However, their behavior is known to become unstable in the face of text inputs containing nonwords that have no definition within a language. This behavior not only results in situations where image generation does not match human expectations but also hinders these models from being utilized in psycholinguistic applications and simulations. This paper exploits the human nature of associating nonwords with their phonetically and phonologically similar words and uses it to propose a T2I generation framework robust against nonword inputs. The framework comprises a phonetics-aware language model as well as an adjusted T2I generation model. Our evaluations confirm that the proposed nonword-to-image generation synthesizes images that depict visual concepts of phonetically similar words more stably than comparative methods. We also assess how the image generation results match human expectations, showing a better agreement than the phonetics-blind baseline.

show abstract

Interpreting Character Embeddings With Perceptual Representations: The Case of Shape, Sound, and Color

Cited by 2 publications

References 0 publications

IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining

IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining

Interpolating the Text-to-Image Correspondence Based on Phonetic and Phonological Similarities for Nonword-to-Image Generation

Contact Info

Product

Resources

About