Benchmarking Drag⋆ for eye direction transformation and beyond

Abstract Eye direction plays a crucial role in determining the quality of photographs containing human faces. Images where subjects look in inconsistent directions are often perceived as low-quality and discarded. While state-of-the-art deep generative models such as DragGAN, DragDiffusion, and DragonDiffusion (collectively referred to as Drag⋆) offer potential solutions for eye direction transformation, their effectiveness for this specific task remains unexplored. In this work, we systematically investigate the capability of Drag ⋆ for eye direction transformation. Our initial experiments reveal that these models in their original form cannot effectively perform this task. To address this limitation, we construct a specialized dataset (i.e., eye multi-direction dataset (EMDD)) and establish a comprehensive benchmark for evaluating methods through fine-tuning on our curated data. Our analysis demonstrates that fine-tuned models achieve satisfactory results when the angular difference between the directions of the source and target eyes is small. However, we observe significant performance degradation when large directional changes are necessary. Through detailed investigation, we uncover the underlying causes of these limitations and provide insights into the models’ failure modes. To overcome these challenges, we propose the edge-localized point selector and zero-latent source region replacement , which can alleviate the identified limitations. Experimental results demonstrate that our approach achieves substantial performance improvements for eye direction transformation, particularly in scenarios involving large angular changes.