A three-tier generative AI workflow for metaverse asset creation

Abstract The Metaverse as a platform for virtual interaction underscores the critical importance of 3D assets in constructing virtual environments. This study proposes a Generative AI-based workflow for Metaverse asset creation by developing three main approaches: (1) 3D asset generation through a text-to-image and image-to-3D pipeline using SDXL, Hunyuan3D-2, Trellis, and TripoSR; (2) texture generation for 3D asset using StableProjectorZ and SDXL; and (3) 3D asset creation via scripts generated by Large Language Models (LLMs) integrated with Blender. The generative AI-assisted workflow is compared against a procedural generation method. Our experiments demonstrate the successful creation and implementation of 3D assets within a Metaverse platform, albeit with limitations such as the necessity for manual intervention and a dependency on the underlying pretrained models. Furthermore, the high complexity of AI-generated objects remains a primary challenge, necessitating a post-processing step to optimize these assets for use in a Metaverse environment. We achieved an average polygon count reduction of 89.29% in AI-generated objects while maintaining visual quality and ensuring usability within the OpenSimulator Metaverse environment. Furthermore, the average similarity to the original objects, measured by chamfer distance, reached 0.027 (L1) and 0.020 (L2). Our study highlights the trade-off of integrating generative AI into the 3D asset creation process, with careful consideration for the critical factors of diversity, quality, complexity, and usability.