Comparing accessibility evaluation tools: a method for tool effectiveness

Abstract

This paper claims that effectiveness of automatic tools for evaluating web site accessibility has to be itself evaluated, given the increasingly important role that these tools play. The paper presents a comparison method for a pair of tools that takes into account correctness, completeness and specificity in supporting the task of assessing the conformance of a web site with respect to established guidelines. The paper presents data acquired during a case study based on comparing LIFT Machine with Bobby. The data acquired from the case study is used to assess the strengths and weaknesses of the comparison method. The conclusion is that even though there is room for improvement of the method, it is already capable of providing accurate and reliable conclusions

Towards Automated Web Accessibility Evaluation: A Comparative Study

“…These data findings slightly differ from previous investigations, e.g. 66% (202) -96% (192) by Vigo et al [31] and even insignificantly increased with those established by G. Brajnik [29], which is between 79% (118) and 93% (139). Yet, all studies indicate to the considerably positive correctness of AWAETs.…”

“…It follows that more than a half number of TP remains unverified if tools are applied separately. Our results for completeness match with [31], which is (14-59%) and are more likely improved when compared with those that ranges between 30% (21TP) and 49% (50TP) [29] as well as between 14% (91TP) and 38% (249TP) [31].…”

“…In our case, although the chosen tools looked for the same and common kinds of problems, we could easily observe the different specificity ratios among them. Notably, TAW as having the poorest specificity result with 164 TP is slightly superior to early tools such as Bobby and LIFT Machine that implemented 70 and 103 automated tests, respectively [29].…”

“…Nevertheless, these three tools turned out to be good assistants. Then, Brajnik (2004) [29] introduced comparison way for tools through three dimensions of software quality such as correctness, completeness and specificity. He found that LIFT Machine and Bobby were already capable of providing accurate and reliable results despite having more room for improvement.…”

“…There are two approaches for organizing sampling data: developing own test pages and using already existing real ones. Creating special test pages is time-consuming and labor-intensive due to the high-level requirements of the standards and web technologies as well as considering wider instances of violations from the real world [29]. The advantages of using real sites are based on the facts that they are built in many different ways and consequently, contain numerous real SC; they may have user-generated content, live and on-demand multimedia elements and frequent updates of content, design, services and features.…”