Do Potential Issues Reported by Automated Tools Affect Guideline-based Web Accessibility Evaluation? A Case Study on Greek Public Hospitals

Software accessibility reveals how easy it is to use an interactive system, regardless of disabilities and devices used. Website accessibility is particularly important due to the popularity of the Web. This is why there are web accessibility standards, such as the Web Content Accessibility Guidelines (WCAG), and technologies available that can help in developing accessible websites. WCAG includes many criteria and techniques that need to be followed to ensure website accessibility. As a result, software tools, often referred to as automated accessibility evaluation tools, have been developed to facilitate this rather demanding task. These tools do, however, report many potential accessibility issues that call for human judgment and thus still require a significant amount of time and effort. Do such potential issues reported by automated tools affect the results of guideline-based web accessibility evaluation? This paper attempts to provide an answer to this question in the context of web accessibility evaluation of Greek public hospitals. To this end, 112 Greek public hospital websites were evaluated against WCAG 2.1 using the TAW automated tool. Results found a significant effect of the manual inspection of potential issues on the accessibility results obtained. The only exception was when the data were analyzed per WCAG priority level where no significant effect was found for level AA. We also build linear regression models to estimate the number of actual accessibility issues based only on the ones that are automatically identified.