Enhancing widget recognition for automated Android testing using Computer Vision techniques

Widget recognition is crucial for automated Android black box testing. Over the past ten years, different industrial tools and academic works have been available for identifying Graphical User Interface (GUI) components in Android screens. Traditional identification methods, like GUI hierarchy parsing, often struggle with dynamic content and complex structures. In contrast, Computer Vision (CV) techniques provide greater robustness and flexibility to adapt to different screen resolutions, design specifications, and patterns. However, the CV-based solutions available are still limited concerning the variety of widgets that can be recognized. Moreover, the current identification of GUI components mainly relies on classification, which can lead to ambiguous lists with repeated elements. In this paper, we combine different CV-based techniques to extract context-based descriptions for each widget, to enhance the identification process by going beyond class recognition for describing widgets. We also implemented two primary CV-based approaches for widget recognition: Object Detection combined with Classification, and a One-Stage Recognition method. We trained and evaluated the approaches on a custom 105 classes widget dataset. Moreover, we present a Computer Vision-based method for describing widgets using their contextual meaning on Android screen captures.