AccessVision: A Screenshot-Only Pipeline for Mobile UI Localization and Visual Accessibility Screening

Mobile application screenshots contain valuable information about user interface geometry, text, colors, and visual elements, though many semantic accessibility properties cannot be extracted from screenshot pixels alone. This paper investigates which accessibility signals can be extracted from mobile application screenshots without access to application source code or runtime accessibility trees. AccessVision detects six types of UI components: buttons, text, images, input fields, icons, and checkboxes. It applies four rule-based visual checks—touch-target size, text-to-background contrast, rendered text height, and proximity of visible labels to associated interface elements—with status-aware reporting traceable to the evaluation procedure. The research prototype is built on a YOLOv8n object detection interface paired with Tesseract OCR for text recognition, geometric calculations for UI component localization, and sRGB relative-luminance calculations for color contrast estimation. Two diagnostic training runs of 10 epochs each were executed on a CPU using weakly supervised development data consisting of 39 screens from four applications. In validation across nine images, Run 1 achieved a precision of 0.0243, recall of 0.2970, [email protected] of 0.0192, and [email protected]:0.95 of 0.00635, while Run 2 achieved a precision of 0.2210, recall of 0.1690, [email protected] of 0.0303, and [email protected]:0.95 of 0.0128. The packaged synthetic development benchmark includes 100 manifest entries, 23 unique screenshots, and 801 UI annotations tested against the four accessibility rules under controlled conditions. This manuscript reports a system-design and preliminary diagnostic study covering the prototype architecture, four rule-based checks, evidence boundaries, and evaluation procedure. Due to the small development dataset and limited annotations, no validated real-world accuracy, peer-review status, or formal WCAG/Android conformance claim is asserted.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24269187
Primary Topic
Digital Accessibility for Disabilities
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

AccessVision: A Screenshot-Only Pipeline for Mobile UI Localization and Visual Accessibility Screening

Aryan Nitin Chavan, Israr Shaikh, Vyom Shinghai, Amey Mode
1 links
Zenodo (CERN European Organization for Nuclear Research)
Digital Accessibility for Disabilities
article

AccessVision: A Screenshot-Only Pipeline for Mobile UI Localization and Visual Accessibility Screening

Aryan Nitin Chavan, Israr Shaikh, Vyom Shinghai, Amey Mode
article en

Abstract

Mobile application screenshots contain valuable information about user interface geometry, text, colors, and visual elements, though many semantic accessibility properties cannot be extracted from screenshot pixels alone. This paper investigates which accessibility signals can be extracted from mobile application screenshots without access to application source code or runtime accessibility trees. AccessVision detects six types of UI components: buttons, text, images, input fields, icons, and checkboxes. It applies four rule-based visual checks—touch-target size, text-to-background contrast, rendered text height, and proximity of visible labels to associated interface elements—with status-aware reporting traceable to the evaluation procedure. The research prototype is built on a YOLOv8n object detection interface paired with Tesseract OCR for text recognition, geometric calculations for UI component localization, and sRGB relative-luminance calculations for color contrast estimation. Two diagnostic training runs of 10 epochs each were executed on a CPU using weakly supervised development data consisting of 39 screens from four applications. In validation across nine images, Run 1 achieved a precision of 0.0243, recall of 0.2970, [email protected] of 0.0192, and [email protected]:0.95 of 0.00635, while Run 2 achieved a precision of 0.2210, recall of 0.1690, [email protected] of 0.0303, and [email protected]:0.95 of 0.0128. The packaged synthetic development benchmark includes 100 manifest entries, 23 unique screenshots, and 801 UI annotations tested against the four accessibility rules under controlled conditions. This manuscript reports a system-design and preliminary diagnostic study covering the prototype architecture, four rule-based checks, evidence boundaries, and evaluation procedure. Due to the small development dataset and limited annotations, no validated real-world accuracy, peer-review status, or formal WCAG/Android conformance claim is asserted.

Zenodo (CERN European Organization for Nuclear Research)
D Y Patil International University (IN)
Digital Accessibility for Disabilities
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon