LLAMA-VG: A Video Vision LLaMA-Based Model for Endoscopy Video Generation

Endoscopy video data is essential for advancing intelligent endoscopic surgery, while constraints arising from privacy regulations and institutional policies significantly limit its collection, hindering the development of large-scale pub-lic datasets. Video generation methods, such as GANs and diffusion models, hold promise in addressing this is-sue by generating synthetic endoscopy videos, but their capacity in modeling complex spatiotemporal correlations remains limited. Accordingly, we propose a novel endoscopy video generation framework LLaMA-VG to incorporate ro-tary positional embeddings and spatiotemporal transformations for long-range temporal reasoning. Besides, we ap-ply multimodal knowledge from pretrained Vision LLaMA to guide the diffusion process, retaining the context sim-ilarity between generated and real videos. We conducted evaluations of our LLaMA-VG on the Colonoscopic and Kvasir-Capsule datasets, in which the model demonstrated considerable improvements over existing state-of-the-art approaches. In particular, LLaMA-VG improves Fréchet Video Distance (FVD) by 19.69% to 22.29%, Fréchet In-ception Distance (FID) by 7.08% to 27.94%, and Inception Score (IS) by 2.31 % to 4.69%. These results underscore the model's ability to generate videos with im-proved temporal coherence and spatial fidelity, simulta-neously maintaining an adequate level of diversity in the generated content. The code is available at: h t t p s: //github.com/YueyaoChenCYY/LLaMA-VG<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.