The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests. In this paper we empirically show the capabilities of current automated evaluation tools. To do so, we investigate the effectiveness of 6 state-of-the-art tools by analysing their coverage, completeness and correctness with regard to WCAG 2.0 conformance. We corroborate that relying on automated tests alone has negative effects and can have undesirable consequences. Coverage is very narrow as, at most, 50% of the success criteria are covered. Similarly, completeness ranges between 14% and 38%; however, some of the tools that exhibit higher completeness scores produce lower correctness scores (66-71%) due to the fact that catching as many violations as possible can lead to an increase in false positives. Therefore, relying on just automated tests entails that 1 of 2 success criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives.

This paper assesses the accessibility of the websites of the National Library of Australia, and those of each of the State/Territory Libraries. The analysis has been conducted using expert manual evaluation, automated tools and users with disabilities to identify both the Web Content Accessibility Guideline (WCAG) Version 2 compliance as well as the chief accessibility barriers identified by users with disabilities. While the results from the different aspects of the hybrid testing methods differ considerably in their ranking, the quantitative data suggest that at the time of writing, none of the libraries assessed meet WCAG 2.0 Level A compliance. Unfortunately, it follows that people with disabilities would have problems accessing materials from the websites of all of the nine libraries tested. In view of the fact that one in five people have a disability that places restrictions on their mobility, employment and/or education, this is understandably significant. Despite the issue of non-compliance, however, many libraries had clearly considered and implemented elements of WCAG 2.0 and would only require minimal improvements to reach the web standard, while others have considerable work to do before they meet the required inclusive website design advocated by both Australian and international standards

No abstract

The Australian Government has implemented the Web Accessibility National Transition Strategy (NTS) which mandates compliance with WCAG 2.0. Those sites not covered by the NTS fall within the purview of the Australian Human Rights guidelines, which recommend WCAG 2.0 AA as a minimum standard. This research will assess this strategy over the period of its implementation as well as building a framework to assist other organisations in their efforts to build more accessible websites.

No abstract