Facilitate and Scale Up the Creation of 3D Meshes, 6D Category-Based Datasets and Grasping with Generative Models: GenVegeFruits3D

Despite significant advances in 2D image, enabled by foundation models, progress in 3D understanding, particularly in 6D pose estimation and shape reconstruction, remains limited by the scarcity of datasets. In particular, in category 6D pose estimation, the high costs of real-world data collection have resulted in datasets with restricted categories, low diversity, and minimal instance variability. Recent methods have attempted to address this gap by leveraging synthetic image generation tools. However, these approaches are constrained by the limitations of available mesh datasets, which hinder the diversity, scalability, and inclusion of novel objects in generated samples. In this work, we propose a first automatic pipeline for the large-scale generation of 3D category-based datasets. Our approach uses 3D generative models guided by textual input to produce diverse and scalable datasets. To demonstrate its efficacy, we generated a new dataset named GenVegeFruits3D comprising 100 categories of fruits and vegetables, each containing over 1000 unique meshes. This significantly enhances the scale and diversity of existing category-based 3D datasets while reducing reliance on pre-existing 3D meshes. Additionally, we trained a 3D generative model, a 3D understanding model, and a grasping model, including on a real robotic setup. The dataset and code are available at: GenVegeFruits3D.