DEArt: Building and evaluating a dataset for object detection and pose classification for European art

• Model with average precision close to state-of-the-art models for other domains, extensively evaluated and compared with SotA models in Computer Vision. Second evaluation of our model compared to models for existing Cultural Heritage datasets. • Manual evaluation of the correctness and relevance of the annotations generated by our model for three datasets from the Europeana collections. • An approach and preliminary experiments to generate new/refined labels for objects and relationships between objects without manual annotation, based on language models. • Discussion of the fairness of our DEArt dataset. Annotation of cultural heritage artefacts allows finding and exploration of items relevant to user needs, supports functionality such as question answering or scene understanding, and in general facilitates the exposure of the society to our history and heritage. But most artefacts lack a description of their visual content due to the assumption that one sees the object; this often means that the annotations effort focuses on the historical and artistic context, information about the painter, or details about the execution and medium. Without a significant body of visual content annotation, machines cannot integrate all this data to allow further analysis, query and inference, and cultural institutions cannot offer advanced functionality to their users and visitors. Given how time-consuming manual annotation is, and to enable the development of new technology and applications for cultural heritage, we have provided through DEArt the most extensive art dataset for object detection and pose classification to date. The current paper extends this work in several ways: (1) we introduce an approach for generating refined object and relationship labels without the need for manual annotations, (2) we compare the performance of our models with the most relevant state-of-the-art in both computer vision and cultural heritage, (3) we evaluate the annotations generated by our object detection model from a user viewpoint, for both correctness and relevance, and (4) we briefly discuss the fairness of our dataset.