Multi-Modal Visual Analytic Approach for Attributing and Authenticating Ancient Chinese Paintings

The traditional workflow of authenticating ancient Chinese paintings heavily relies on experts’ reasoning and justification, who need to consider various information, such as inscriptions, seals, and painting content, along with extensive literature materials. Although deep learning-based methods have improved the efficiency of traditional authentication workflow, they predominantly focus on image analysis while the black-box nature of DNNs hampering the effective human-machine collaboration. To address this issue, we propose a visual analysis approach that extract multi-modal semantics to facilitate the easy analysis of ancient Chinese paintings. Firstly, ancient painting images are segmented into three semantic objects: inscriptions, seals, and painting content. Then, an association structure is constructed between the corresponding segments of ancient paintings through image matching and text recognition, as well as with relevant literary references. Finally, experts select segments of ancient paintings to generate a multi-modal semantic network graph for reference and verification. Furthermore, we develop a mixed-initiative system for assisting the visual attribution analysis of ancient Chinese paintings. Through case studies and expert evaluations, the effectiveness and accuracy of ancient painting analysis and authentication were validated using a 5-point Likert scale for expert feedback, with scores ranging from 0.78 to 1.78.