HybridDeform4D: Joint Refinement of Mesh and Gaussian Splatting for Video-to-4D Object Generation

Generating dynamic, spatiotemporally consistent 3D content from monocular video is a significant challenge. Existing methods leveraging Gaussian Splatting (GS) often struggle to maintain such coherence without strong regularization. We introduce HybridDeform4D, a novel framework that enhances 4D object generation by jointly optimizing a deformable mesh with its surface-bound GS attributes. This framework supports starting with a coarse mesh and capturing both complex motion and appearance variations without requiring high-quality initialization. To ensure coherent animation, we introduce a coarse-to-fine optimization scheme. This strategy first focuses on learning motion around a key reference frame before progressively expanding to optimize the entire video sequence, ensuring stable and accurate temporal dynamics. Experimental results show that HybridDeform4D achieves high rendering quality and spatial-temporal consistency. Furthermore, we demonstrate the versatility of our approach by extending it to text-to-3D synthesis.