AR and ML Solutions for Facial Expression Recognition on Mobile Phones

The widespread use of smartphones has led to the development of a diverse range of personal and business applications. Especially, ever-improving computational power of smartphones has made real-time video processing operations feasible on mobile and portable devices. Consequently, real-time facial expression recognition (FER) on mobile phones has emerged as an efficient approach, despite certain limitations imposed by hardware and power storage limitations. An example of such implementation is the ubiquitous "Face Unlock" feature utilized in numerous smartphones and applications, including online banking. Facial expression recognition that only relies on Machine Learning (ML) faces challenges; such as the requirement for substantial datasets and expertise-level training to achieve accurate models. Accordingly, this study explores an alternative approach which involves Augmented Reality (AR) to achieve expression recognition. To achieve this goal, this study undertakes a comprehensive comparison of these two distinct techniques through the development of separate mobile applications for facial expression recognition. Experimental results indicate that the ML-based architecture achieves the highest classification accuracy of 91.63%, outperforming the AR-based approach (81.63%). However, from an operational efficiency viewpoint, AR-based approach demonstrates superior performance with an average inference latency of 37.8 ms, operating approximately 7.6 times faster than ML application (288.2 ms). Furthermore, environmental stress testing reveals that while RGB-based ML models suffer performance degradation in low light, the AR system maintains consistent tracking in total darkness due to active infrared depth sensing. The findings shed light on the advantages and limitations of each method, contributing to the advancement of FER technology in mobile applications.