AccessMenu: Enhancing Usability of Online Restaurant Menus for Screen Reader Users

Online food ordering has become commonplace due to its convenience. The wide variety of culinary choices, combined with fast and economical door-delivery services, encourages more people to order food online. To facilitate this process, food vendors, including restaurants, often provide full menus on their websites, typically in visual formats such as images or PDFs. While this is convenient for sighted users, blind and visually impaired (BVI) individuals face significant challenges accessing these visual menus with their screen reader assistive technology. An interview study with 12 BVI screen reader users revealed that present assistive tools do not adequately satisfy the needs of these users, with issues ranging from text-ordering errors, to inaccurate inferences (e.g., incorrectly categorizing a Caesar salad with anchovies as vegetarian), to misinterpretation of symbols and legends. Moreover, the users expressed a need for a screen reader-tailored interface to access the information in menus. To address these access barriers and users’ needs, we present AccessMenu, a browser extension that automatically detects visual menus in restaurant websites, uses multi-modal large language models to extract and analyze the menu content, and re-renders it in a conveniently navigable HTML format accessible with screen readers. AccessMenu also enables BVI users to issue natural language queries, allowing them to efficiently distill specific information from the menus. In a user evaluation with 10 blind participants, AccessMenu significantly outperformed a state-of-the-art solution in usability and task workload, by providing convenient menu navigation and query-based menu filtering capabilities.