CAD-RAG: A multi-modal retrieval augmented framework for user editable 3D CAD model generation

Computer-Aided Design (CAD) has revolutionized design and manufacturing by enabling precise, complex models in collaborative environments.While similar CAD models with application-specific modifications are often required, designs are typically created from scratch due to challenges in retrieving existing models or generating editable ones.Although parametric CAD modeling has advanced through deep generative approaches treating CAD as a language task to generate user-editable designs, building truly scalable multi-modal datasets and networks tailored for 3D design tasks, particularly in engineering domains remains a significant challenge.Developing such datasets, especially those incorporating images, point clouds and user-like text and hand-drawn sketches is difficult as these modalities demand fine-grained geometric understanding and extensive human-in-the-loop evaluations.While large foundational models like CLIP have improved cross-modal retrieval, they are primarily trained on natural images and fail to capture the geometric and structural complexities inherent to CAD data.In this paper, we propose a novel multi-modal pipeline for CAD command sequence generation using state-of-theart Vision-Language Models (VLMs).We introduce a unique multimodal CAD dataset comprising hand-drawn sketches, CAD command sequences, images and basic text prompts.These modalities are integrated through a Multi-modal Retrieval-Augmented Generation (MM-RAG) framework to enable user-editable CAD model retrieval and generation.Our RAG-based pipeline streamlines the CAD design process by enabling iterative, userguided model generation based on simple sketches or text queries.This approach aims to streamline CAD model design by creating an advanced, end-to-end pipeline that supports design workflows.The dataset and code will be made publicly available at: https://github.com/ananthu2014/cadrag.