Coverage of web accessibility guidelines provided by automated checking tools

Abstract Within the European Union and other jurisdictions, government agencies are required to make digital offerings like websites, apps, or documents accessible such as by following the recommendations of the Web Content Accessibility Guidelines (WCAG). Using websites of Swedish public sector organizations and automated accessibility checking tools, we both assess to which degree those websites achieve the legal requirements and which limitations the used tools have. Our results show that the median number of unique violated success criteria is only two or less for most tools, but the tools cover only one sixth of all WCAG success criteria. Tools differ in their assessments due to several reasons, such as the size and complexity but also inconsistency of WCAG and related documentation (‘techniques’), and typical software engineering problems in the corresponding implementations. To improve the coverage and quality of tools we suggest the incorporation of scientific knowledge, for example on readability testing, and the creation of a benchmark suite similar to those in related fields like PDF/A validation.