Towards consistent sketch-guided local 3D shape editing

• We developed a diffusion-based generative framework called SEN, designed for sketch-guided local shape editing. This framework ensures consistency in geometric coherence and preservation. • We introduced an attention fusion module that effectively connects the cross-modal conditions to produce the final shape editing results. • We developed a sketch generator to directly generate sketch images from point clouds to facilitate our SEN training. Modeling complex 3D shapes from 2D sketch images has gained significant attention due to recent advancements in deep generative models. While many efforts have focused on producing geometries that faithfully reflect sketches, there has been limited exploration of sketch-guided local editing. Existing shape generation techniques often modify not only the intended edited areas but also the unedited parts, compromising geometric coherence and fairness among different parts of the shape. This paper introduces a shape editing network (SEN) under the diffusion-based generative framework to specifically enhance consistency in sketch-guided local shape editing. Our core insight is capturing partial geometric structure correlations to achieve consistent editing results. To accomplish this, the SEN is designed to conditionally learn local edits of a partial shape while considering the geometries of the unedited parts and the sketch image. We also devise an attention fusion module that progressively blends and aligns cross-modal conditioning streams between the latent variables of the sketch and point cloud representations. Additionally, by incorporating a guided sampling strategy for diffusion modeling, our SEN allows for more flexible control, even with complex or limited conditioning patterns. Our method significantly improves the geometric coherence of the synthesized parts while preserving the shapes of unedited areas. We demonstrate the advantages of our approach qualitatively and quantitatively across various shape categories, outperforming state-of-the-art methodologies.