Fouille d'images par synthèse

Image archives contain lots of hidden knowledge that researchers in the digital humanities would like to discover at scale. Much of this knowledge is hard to described through textual descriptions, or to properly ground them on visual evidence. Given collections of images with general predefined classification, for example the name of family of letters or that of a country, the goal of this thesis is to develop machine learning approaches that can mine informative visual structure hiding behind these image collections. Recent compositional synthesis methods can incorporate weak human labels to unsupervised approaches. The key idea of this thesis is to repurpose them towards reliable human-interpretable visual summarization, to provide a method that automatically identifies and summarizes informative visual structure. Our work focuses on two important challenges,The first challenge, is to summarize and help refine the typology existing typologies of handwritten characters. Character morphology has been central to the field of palaeography, where existing typologies are described through textual descriptions. The reliance of the literature on such descriptions prohibits a quantitative methodology that could ground qualitative observation and exploration across historical manuscript collections. Our first contribution, the "Learnable Typewriter," is a novel approach that learns an explicit decomposition of the script of a manuscript's text lines, into small images called sprites. Learning to reconstruct input text lines by composing sprites via differentiable transformations using weak-supervision, our method produces a concrete, interpretable summarization of a character’s morphology. This enables palaeographers to perform quantitative comparison that stands very close to the concrete visual evidence, captured through sprites. We validate this approach on various datasets of printed modern and historical fonts and ciphered texts, and two case studies on medieval manuscripts.The second challenge, is to extract and define the elements that best support a visual categorization of images in the context of their datasets. Different archives of historical, or cultural image datasets, are often associated with time or location, however finding their “visual genome” is hard to define through text and enumerate. In our second contribution, "Diffusion Models as Data Mining Tools", we leverage the abstract and scalable compositional synthesis capabilities of diffusion models to mine visual structure across versatile weakly labeled datasets. Through a "typicality" score that is based on the influence of a label to the denoising performance of the diffusion model, we rank and then cluster elements typical to specific labels. Unlike other approaches that rely on pairwise matching to discovery both frequent and discriminative patches, our diffusion based approach is linear, has more robust representations and can scale to versatile datasets of cars, portraits, geographical images, or scenes, that consist from thousands to millions of images. Using the diffusion model, we can directly translate images across location and mine co-typical, as well as make the model's sampling bias more interpretable.