A real-time mobile solution for shoe try-on using foot pose estimation and 3D processing techniques

Implementing Augmented Reality (AR) in virtual try-on technology has revolutionized the online shopping experience, transforming how clients engage with products. This technology allows customers to try on clothes without direct physical contact, which has become convenient and valuable in the age of online shopping. Despite clothing being a dominant sector, shoes also hold a significant portion of the market. However, there has been limited previous research conducted on the subject of virtual shoe try-on. This paper presents an innovative solution to the Foot Pose Estimation problem, offering a deep learning model that delivers accurate results while operating in real-time on the CPU. Due to the insufficiency of public foot keypoint datasets, a medium-scale self-collected 2D foot keypoints dataset has been conducted with 9 keypoints each foot for training and evaluating the model. In addition, this model has successfully been utilized to create a shoe AR try-on application for smartphones. It provides a solution that generates a realistic 3D shoe model for a smooth and stable try-on experience. Practical tests have proven that the system gives real-time performance under mobile computing conditions.