Testability and validity of WCAG 2.0

Abstract

Web Content Accessibility Guidelines 2.0 (WCAG 2.0) require that success criteria be tested by human inspection. Further, testability of WCAG 2.0 criteria is achieved if 80% of knowledgeable inspec tors agree that the criteria has been met or not. In this paper we in vestigate the very core WCAG 2.0, being their ability to determine web content accessibility conformance. We conducted an empir ical study to ascertain the testability of WCAG 2.0 success crite ria when experts and non-experts evaluated four relatively complex web pages; and the differences between the two. Further, we dis cuss the validity of the evaluations generated by these inspectors and look at the differences in validity due to expertise.In summary, our study, comprising 22 experts and 27 non-experts, shows that approximately 50% of success criteria fail to meet the 80% agreement threshold; experts produce 20% false positives and miss 32% of the true problems. We also compared the performance of experts against that of non-experts and found that agreement for the non-experts dropped by 6%, false positives reach 42% and false negatives 49%. This suggests that in many cases WCAG 2.0 con formance cannot be tested by human inspection to a level where it is believed that at least 80% of knowledgeable human evaluators would agree on the conclusion. Why experts fail to meet the 80% threshold and what can be done to help achieve this level are the subjects of further investigation.

Understanding Accessibility as a Process through the Analysis of Feedback from Disabled Students

“…As key words may reflect specific types of issues, these can be categorised, forming dictionaries of related terms. Comments express meanings that are not fully comprehensible through an individual word or phrase, so thematic analysis offers a means to identify and interpret themes and code comments according to these [3].…”

“…The identified key words were also the subject of a manual thematic analysis [3]. This approach supported the identification of emergent themes and meanings through interpreting comments in full.…”

“…For example, Power et al (2012) found that only 50.4% of problems encountered by blind users in evaluations were covered by WCAG 2.0 [16]. Brajnik et al (2010) found that even expert evaluators could find it difficult to reliably identify problems in websites using WCAG 2.0, or to reach shared agreement on the problems that existed [3].…”

Web Accessibility Challenges

“…They found that they were unclear on the differences between automated and manual testing of accessibility and what can be tested through the automated tools. In addition, [38] had 22 expert and 27 non-expert evaluators to perform accessibility evaluation using WCAG 2.0 and discovered that 50% of testers were unable to come to an 80% level of agreement about whether a problem was present in a webpage. Additionally, when using WCAG 2.0 20% of the problems reported by the evaluators were false positives and 32% of the true accessibility problems had been missed.…”

“…Different AEMs lead to different kinds of results and quality. They require different levels of resources and differ in their applicability [38]. These methods have contrasting pros and cons and various properties [52].…”