AccessVision: A Screenshot-Only Pipeline for Mobile UI Localization and Visual Accessibility Screening
Mobile application screenshots contain valuable information about user interface geometry, text, colors, and visual elements, though many semantic accessibility properties cannot be extracted from screenshot pixels alone. This paper investigates which accessibility signals can be extracted from mobile application screenshots without access to application source code or runtime accessibility trees. AccessVision detects six types of UI components: buttons, text, images, input fields, icons, and checkboxes. It applies four rule-based visual checks—touch-target size, text-to-background contrast, rendered text height, and proximity of visible labels to associated interface elements—with status-aware reporting traceable to the evaluation procedure. The research prototype is built on a YOLOv8n object detection interface paired with Tesseract OCR for text recognition, geometric calculations for UI component localization, and sRGB relative-luminance calculations for color contrast estimation. Two diagnostic training runs of 10 epochs each were executed on a CPU using weakly supervised development data consisting of 39 screens from four applications. In validation across nine images, Run 1 achieved a precision of 0.0243, recall of 0.2970, [email protected] of 0.0192, and [email protected]:0.95 of 0.00635, while Run 2 achieved a precision of 0.2210, recall of 0.1690, [email protected] of 0.0303, and [email protected]:0.95 of 0.0128. The packaged synthetic development benchmark includes 100 manifest entries, 23 unique screenshots, and 801 UI annotations tested against the four accessibility rules under controlled conditions. This manuscript reports a system-design and preliminary diagnostic study covering the prototype architecture, four rule-based checks, evidence boundaries, and evaluation procedure. Due to the small development dataset and limited annotations, no validated real-world accuracy, peer-review status, or formal WCAG/Android conformance claim is asserted.
Authors
- Aryan Nitin Chavan
- Israr Shaikh
- Vyom Shinghai
- Amey Mode
Institutions
- D Y Patil International University (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24269187
- Primary Topic
- Digital Accessibility for Disabilities
- Type
- article
- Field-Weighted Citation Impact
- 0.00