Measuring Human Readability of Machine Generated Text: Three Case Studies in Speech Recognition and Machine Translation

Jones, Douglas A.; Gibson, Edward; Shen, Wade; Granoien, Neil; Herzog, Martha; Reynolds, Douglas A.; Weinstein, Clifford J.

doi:10.1109/icassp.2005.1416477

Cited by 21 publications

(15 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Usefulness is also examined by Roturier (2006) alongside comprehension and acceptability as rated by the user. Similarly, Jones et al (2005) examine the readability of the output generated by an MT system by means of measuring accuracy in answering questions, task time, and a subjective rating given by users. Lastly, Gaspari (2004) evaluated users' needs when using online MT systems in terms of guessability (the user's expectations when first using an online system) and learnability (the capacity of the system to enable a user to learn how to use it), where the focus rested on the systems themselves.…”

Section: Introductionmentioning

confidence: 99%

Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

Doherty

O’Brien

2013

International Journal of Human-Computer Interaction

View full text Add to dashboard Cite

This paper reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centred approach, we utilize the ISO/TR 16982 definition of usability -goal completion, satisfaction, effectiveness, and efficiency -and apply eye-tracking measures shown to be reliable indicators of cognitive effort, along with a post-task questionnaire. We investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, we found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw 2 machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English.

show abstract

Section: Introductionmentioning

confidence: 99%

Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

Doherty

O’Brien

2013

International Journal of Human-Computer Interaction

View full text Add to dashboard Cite

show abstract

“…We then combined these objective and subjective ratings to create a Rater Readability factor score. We followed Jones et al (2005), who argue that readability is comprised of: (a) comprehension; (b) time to complete and answer questions about the reading; and (c) how an individual subjectively perceives the text. We estimated an exploratory factor analysis model with six variables.…”

Section: Validating the Opinion Clarity Measurementioning

confidence: 99%

The Influence of Public Sentiment on Supreme Court Opinion Clarity

Black¹,

Owens²,

Wedeking³

et al. 2016

Law Soc Rev

View full text Add to dashboard Cite

We examine whether public opinion leads Supreme Court justices to alter the content of their opinions. We argue that when justices anticipate public opposition to their decisions, they write clearer opinions. We develop a novel measure of opinion clarity based on multifaceted textual readability scores, which we validate using human raters. We examine an aggregate time series analysis of the influence of public mood on opinion clarity and an individual‐level sample of Supreme Court cases paired with issue‐specific public opinion polls. The empirical results from both models show that justices write clearer opinions when their rulings contradict popular sentiment. These results suggest public opinion influences the Court, and suggest that future scholarship should analyze how public opinion influences the written content of decision makers’ policies.

show abstract

“…Yet, studies have shown that even though current Machine Translation technology is not good enough to provide final professionalquality translation, it is sufficiently good to support gisting ( [17], [8]). In the present situation, the task might even be easier because, in many cases, the machine will only be translating small changes to a page.…”

Section: Dealing With More Than Two Languagesmentioning

confidence: 99%

Translation the Wiki way

Désilets

González²,

Paquet³

et al. 2006

Proceedings of the 2006 International Symposium on Wikis

View full text Add to dashboard Cite

This paper discusses the design and implementation of processes and tools to support the collaborative creation and maintenance of multilingual wiki content. A wiki is a website where a large number of participants are allowed to create and modify content using their Web browser. This simple concept has revolutionized collaborative authoring on the web, enabling among others, the creation of Wikipedia, the world's largest online encyclopedia. On many of the largest and highest profile wiki sites, content needs to be provided in more than one language. Yet, current wiki engines do not support the efficient creation and maintenance of such content. Consequently, most wiki sites deal with the issue of multilingualism by spawning a separate and independent site for each language. This approach leads to much wasted effort since the same content must be researched, tracked and written from scratch for every language. In this paper, we investigate what features could be implemented in wiki engines in order to deal more effectively with multilingual content. We look at how multilingual content is currently managed in more traditional industrial contexts, and show how this approach is not appropriate in a wiki world. We then describe the results of a User-Centered Design exercise performed to explore what a multilingual wiki engine should look like from the point of view of its various end users. We describe a partial implementation of those requirements in our own wiki engine (LizzyWiki), to deal with the special case of bilingual sites. We also discuss how this simple implementation could be extended to provide even more sophisticated features, and in particular, to support the general case of a site with more than two languages. Finally, even though the paper focuses primarily on multilingual content in a wiki context, we argue that translating in this "Wiki Way", may also be useful in some traditional industrial settings, as a way of dealing better with the fast and ever-changing nature of our modern internet world.

show abstract

Measuring Human Readability of Machine Generated Text: Three Case Studies in Speech Recognition and Machine Translation

Cited by 21 publications

References 3 publications

Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

The Influence of Public Sentiment on Supreme Court Opinion Clarity

Translation the Wiki way

Contact Info

Product

Resources

About