Audio-to-Visual Conversion Using Hidden Markov Models

Lee, Soon Kyu; Yook, Dongsuk

doi:10.1007/3-540-45683-x_60

Cited by 33 publications

(44 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We use the concept of a viseme which is the lip shape during the voicing of a phoneme. Here, we use a phoneme to viseme mapping resulting in 14 visemes from the 42 English phonemes [10], [12]. Data corresponding to each viseme are grouped using the available phoneme-level transcriptions.…”

Section: Face Modalitymentioning

confidence: 99%

Decision level combination of multiple modalities for recognition and analysis of emotional expression

Metallinou

Lee

Narayanan

2010

2010 IEEE International Conference on Acoustics, Speech and Signal Processing

View full text Add to dashboard Cite

Emotion is expressed and perceived through multiple modalities. In this work, we model face, voice and head movement cues for emotion recognition and we fuse classi ers using a Bayesian framework. The facial classi er is the best performing followed by the voice and head classi ers and the multiple modalities seem to carry complementary information, especially for happiness. Decision fusion signi cantly increases the average total unweighted accuracy, from 55% to about 62%. Overall, we achieve average accuracy on the order of 65-75% for emotional states and 30-40% for neutral state using a large multi-speaker, multimodal database. Performance analysis for the case of anger and neutrality suggests a positive correlation between the number of classi ers that performed well and the perceptual salience of the expressed emotion.

show abstract

Section: Face Modalitymentioning

confidence: 99%

Decision level combination of multiple modalities for recognition and analysis of emotional expression

Metallinou

Lee

Narayanan

2010

2010 IEEE International Conference on Acoustics, Speech and Signal Processing

View full text Add to dashboard Cite

show abstract

“…This reduction in the number of units reduces the number of possible bigrams of units by about 85% compared with using phone units, and hence reduces VLID performance. Our experience using visemes that were defined by the mapping described in [34] showed that performance was limited, and analysis of this mapping showed that it was highly over-simplified. Although we know that there are several phonemes that cannot be discriminated visually (for instance, it is impossible to detect voicing visually, or place of articulation when this is far back inside the oral cavity), we have found that VLID accuracy is enhanced by training models using video segments corresponding to 42 audio phonemes.…”

Section: B Visual Models Of Phonemesmentioning

confidence: 99%

Language Identification Using Visual Features

Newman

Cox

2012

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Abstract-Automatic visual language identification (VLID) is the technology of using information derived from the visual appearance and movement of the speech articulators to identify the language being spoken, without the use of any audio information. This technique for language identification (LID) is useful in situations in which conventional audio processing is ineffective (very noisy environments), or impossible (no audio signal is available). Research in this field is also beneficial in the related field of automatic lip-reading. This paper introduces several methods for visual language identification (VLID). They are based upon audio LID techniques, which exploit language phonology and phonotactics to discriminate languages. We show that VLID is possible in a speaker-dependent mode by discriminating different languages spoken by an individual, and we then extend the technique to speaker-independent operation, taking pains to ensure that discrimination is not due to artefacts, either visual (e.g. skin-tone) or audio (e.g. rate of speaking). Although the low accuracy of visual speech recognition currently limits the performance of VLID, we can obtain an error-rate of < 10% in discriminating between Arabic and English on 19 speakers and using about 30s of visual speech.

show abstract

“…As previously, for isolated words, the Disney vowels are significantly worse than all others when paired with all consonant difference over the whole group. The Lee [71], Montgomery [67] and Bozkurt [69] vowels are consistently above the mean and above the upper error bar for Disney [66], Jeffers [70] and Hazen [19] vowels. In comparing the consonants, Lee [71] and Hazen [19] are the best whereas Woodward [75] and Franks [62] are the bottom performers.…”

Section: Comparison Of Current Phoneme-to-viseme Mapsmentioning

confidence: 95%

“…Data-driven methods are most recent, e.g. Lee's [71] visemes were presented in 2002 and Hazen's [19] in 2004. The remaining visemes are based around linguistic/phonemic rules.…”

Section: Comparison Of Phoneme-to-viseme Mappingsmentioning

confidence: 99%

“…As we move to more realistic data , continuous speech, many of the datadriven approaches degrade which implies that they data used to derive these visemes was unrealistic. For example the Lee visemes [71] were derived without any use of video data at all so it is hardly surprising that they are fragile when presented with more realistic data.…”

Section: Comparison Of Current Phoneme-to-viseme Mapsmentioning

confidence: 99%

See 1 more Smart Citation

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Bear

Harvey

2017

Speech Communication

View full text Add to dashboard Cite

Visemes are the visual equivalent of phonemes. Although not precisely defined, a working definition of a viseme is “a set of phonemes which have identical appearance on the lips”. Therefore a phoneme falls into one viseme class but a viseme may represent many phonemes: a many to one mapping. This mapping introduces ambiguity between phonemes when using viseme classifiers. Not only is this ambiguity damaging to the performance of audio-visual classifiers operating on real expressive speech, there is also considerable choice between possible mappings. In this paper we explore the issue of this choice of viseme-to-phoneme map. We show that there is definite difference in performance between viseme-to-phoneme mappings and explore why some maps appear to work better than others. We also devise a new algorithm for constructing phoneme-to-viseme mappings from labeled speech data. These new visemes, ‘Bear’ visemes, are shown to perform better than previously known units

show abstract

Audio-to-Visual Conversion Using Hidden Markov Models

Cited by 33 publications

References 9 publications

Decision level combination of multiple modalities for recognition and analysis of emotional expression

Decision level combination of multiple modalities for recognition and analysis of emotional expression

Language Identification Using Visual Features

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Contact Info

Product

Resources

About