Evaluation of Speech Synthesis

Campbell, Nick

doi:10.1007/978-1-4020-5817-2_2

Cited by 12 publications

(9 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, correct prosody and great naturalness are essential in most multimedia applications. The evaluation can be carried out at different levels (segment, word, sentence or paragraph) and with different kinds of tests (Campbell 2007).…”

Section: Experiments and Resultsmentioning

confidence: 99%

Scientific Advisory Board

2008

Journal of Endodontics

View full text Add to dashboard Cite

Quaderns del CAC is a half-yearly electronic journal published in June and December in Catalan, Spanish and English. Paper edition was published from issue 1 (any 1998) to issue 35 (December 2010).The journal wants to motivate critical analysis and debate on culture and media. Edited by the Catalan Audiovisual Council, is intended as a platform for approaching to the field of audiovisual media and culture from multidisciplinary perspectives and it gives a special attention to the European and international research framework. The issues opens with the invited article, followed by the monographic section that analyzes a subject form different approaches.The section "Articles" consists of original and unpublished articles accepted after their submission to a double-blind peer review process, after an initial assessment of quality by the Editorial board.The journal undertakes to respond the authors of the positive or negative result of the process in less than a month and a half after the submission of the article.The manuscript submission guidelines for articles and book reviews are at the final pages of every issue.The opinions expressed in the articles published in Quaderns del CAC are those of the authors and do not necessarily reflected the journal position.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Scientific Advisory Board

2008

Journal of Endodontics

View full text Add to dashboard Cite

show abstract

“…Although evaluations for naturalness and intelligibility had become standard by this point, a 2007 book chapter [20] predicts that tone of voice, manner of speaking, emotional expressiveness, and, generally speaking, "interpersonal skills" will become more important for speech synthesis in the future, and so we will need to find ways to evaluate these aspects as well.…”

Section: Mid-1990s and 2000s: Naturalness Intelligibility And Efforts...mentioning

confidence: 99%

A review on subjective and objective evaluation of synthetic speech

Cooper,

Huang,

Tsao

et al. 2024

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

Evaluating synthetic speech generated by machines is a complicated process, as it involves judging along multiple dimensions including naturalness, intelligibility, and whether the intended purpose is fulfilled. While subjective listening tests conducted with human participants have been the gold standard for synthetic speech evaluation, its costly process design has also motivated the development of automated objective evaluation protocols. In this review, we first provide a historical view of listening test methodologies, from early in-lab comprehension tests to recent large-scale crowdsourcing mean opinion score (MOS) tests. We then recap the development of automatic measures, ranging from signal-based metrics to model-based approaches that utilize deep neural networks or even the latest self-supervised learning techniques. We also describe the VoiceMOS Challenge series, a scientific event we founded that aims to promote the development of data-driven synthetic speech evaluation. Finally, we provide insights into unsolved issues in this field as well as future prospective. This review is expected to serve as an entry point for early academic researchers to enrich their knowledge in this field, as well as speech synthesis practitioners to catch up on the latest developments.

show abstract

“…The recommended standard maintained by the International Telecommunication Union is the mean opinion score (MOS) [64]. Raters give an objective assessment of speech quality on a 5-point scale which involves different areas of comprehension (how easy was it to understand the spoken sentence), naturalness, intelligibility (how well does a transcription by the raters match the original written sentence) and likeability [65]. Both objective as subjective measures are used and quantified, allowing for statistical comparison.This includes design of a measure that is robust to increasing complexity that can carry across different context [66].…”

Section: Recommendations For Gesture Evaluationmentioning

confidence: 99%

A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents

Wolfert,

Robinson,

Belpaeme

2021

Preprint

View full text Add to dashboard Cite

Embodied Conversational Agents (ECA) take on different forms, including virtual avatars or physical agents, such as a humanoid robot. ECAs are often designed to produce nonverbal behaviour to complement or enhance its verbal communication. One form of nonverbal behaviour is co-speech gesturing, which involves movements that the agent makes with its arms and hands that is paired with verbal communication. Co-speech gestures for ECAs can be created using different generation methods, such as rule-based and data-driven processes. However, reports on gesture generation methods use a variety of evaluation measures, which hinders comparison. To address this, we conducted a systematic review on co-speech gesture generation methods for iconic, metaphoric, deictic or beat gestures, including their evaluation methods. We reviewed 22 studies that had an ECA with a human-like upper body that used co-speech gesturing in a social human-agent interaction, including a user study to evaluate its performance. We found most studies used a within-subject design and relied on a form of subjective evaluation, but lacked a systematic approach. Overall, methodological quality was low-to-moderate and few systematic conclusions could be drawn. We argue that the field requires rigorous and uniform tools for the evaluation of co-speech gesture systems. We have proposed recommendations for future empirical evaluation, including standardised phrases and test scenarios to test generative models. We have proposed a research checklist that can be used to report relevant information for the evaluation of generative models as well as to evaluate co-speech gesture use.

show abstract

Evaluation of Speech Synthesis

Cited by 12 publications

References 66 publications

Scientific Advisory Board

Scientific Advisory Board

A review on subjective and objective evaluation of synthetic speech

A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents

Contact Info

Product

Resources

About