Otta: One-Shot Video Tuning with Temporal Adapters on Diffusion

In recent years, with the maturity of text-to-image (T2I) technology, the field of text-to-video (T2V) generation has attracted increasing attention. Numerous related approaches have been proposed; however, existing research either trains models from scratch or crudely incorporates temporal information into pre-trained T2I models, both requiring substantial computational resources. To address this, a framework is proposed that finetunes only a small number of low-rank adaptation (LoRA) layers and lightweight temporal adapters. Leveraging the ability of T2I models to generate static images representing verb semantics, a single generated image is expanded into a sequence of frames to form a video. To ensure motion consistency, a temporally aware attention mechanism and a temporal consistency loss are introduced, where abrupt changes are penalized by the mean squared error of inter-frame differences. During inference, the latent encoding of the first frame is obtained via DDIM inversion [19] and fixed to guide subsequent generations, thereby improving temporal stability.