The Architecture of Visual Foundation Models: A Comprehensive Review of Image, Video, and Multimodal Understanding

Research in visual intelligence has shifted from task-specific convolutional networks trained on curated labels toward large-scale pretrained architectures that transfer across image, video, and language tasks with minimal fine-tuning. Existing surveys typically treat image, video, and multimodal foundation models as separate literatures, or compare them mainly by topic breadth and model count. This review instead reads the three literatures jointly against three explicit research questions concerning architectural convergence, efficiency-versus-scale trade-offs, and the cross-modal diffusion of techniques, and documents its literature-selection procedure with explicit retention criteria and reports the composition of the retained literature. Representative systems analyzed include the Vision Transformer, CLIP, the Segment Anything family (through SAM 3), DINOv3, masked and joint-embedding predictive models (through V-JEPA 2), Flamingo, BLIP-2, LLaVA, Chameleon, Qwen2.5-VL, and GPT-4V, consolidated into ten tables and eight original figures, including a literature-selection flow diagram, a comparison of learning paradigms, a cross-modal milestone timeline, and mechanism-level diagrams of video and vision-language architectures. Four findings emerge: self-supervised paradigms progress through successive, traceable fixes to specific predecessor limitations; connector- and distillation-based efficiency interventions recover performance otherwise requiring substantially more parameters; the early- versus late-fusion divergence in multimodal design reflects an economic rather than a purely technical trade-off; and, within the sample examined, techniques tend to appear in image models before reaching video and multimodal systems, a pattern presented here as a testable hypothesis that 2025 releases both support (V-JEPA 2) and qualify (SAM 3, Qwen2.5-VL). The review closes by identifying open problems in temporal reasoning, hallucination, computational cost, and evaluation validity, and outlines directions likely to shape subsequent research.