Représentations visuelles structurées avec modèles de fondations

Foundation models trained on large-scale datasets - particularly those with strong zero-shot capabilities - offer a promising path toward reducing reliance on annotated data. Among these, diffusion-based generative models have emerged as especially attractive due to their ability to synthesize high-quality images from natural language prompts. Unlike discriminative encoders, diffusion models are natively trained at scale with a stable generative objective and are inherently aligned with textual inputs, making them strong candidates for unifying vision and language under a single training framework. These properties suggest that such models may internally encode rich, semantically structured representations with potential far beyond generation alone. Yet, the potential of these models for structured visual understanding remains largely untapped. One open question is whether their generative mechanisms can be repurposed for temporally coherent video manipulation, a task that demands consistency across frames and alignment with textual instructions. While diffusion models have achieved remarkable results in image editing, extending these capabilities to video remains a significant challenge. Existing approaches typically rely on heavy architectural modifications or supervision across frames, limiting their adaptability. Demonstrating that temporal structure and semantic control can emerge from a frozen image-based diffusion model would represent a critical step toward general-purpose video editing systems. Another underexplored direction is the use of internal diffusion representations for dense prediction tasks, such as semantic segmentation. Unlike encoders explicitly trained for discrimination, diffusion models are not optimized for category separation or fine-grained localization. Nonetheless, their multi-scale U-Net architectures and rich generative training may still encode latent spatial hierarchies that can be leveraged with minimal supervision. Understanding how to extract and interpret these features for zero-shot segmentation opens new avenues for reusing generative models as general-purpose visual backbones. Finally, a persistent limitation of diffusion models and foundation models more globally, lies in their lack of spatial granularity. Architectural choices such as aggressive downsampling or the use of latent spaces often lead to coarse internal representations that fall short in tasks requiring high-resolution reasoning—such as part segmentation, object manipulation, or detailed visual understanding. Overcoming this limitation involves rethinking how feature detail is preserved and refined, possibly through attention mechanisms or architectural augmentations that enhance spatial fidelity without sacrificing semantic richness. Taken together, these challenges define the scientific scope of this thesis: exploring how generative diffusion models—originally designed for image synthesis—can be reinterpreted, augmented, and repurposed to address core problems in zero-shot video editing, dense prediction, and fine-grained visual reasoning. Addressing these questions not only advances our understanding of diffusion-based models but also aligns closely with the practical needs of robust, embedded AI systems in industrial applications.