Robust Differentiable Sketch Rendering for Single‐View 3D Reconstruction

Abstract In this paper, we propose a novel end‐to‐end method to model 3D objects with geometric details from a single‐view sketch input. Specifically, a novel deep learning‐based differentiable sketch renderer is introduced to establish the relationship between geometric features, represented by normal maps, and 2D sketch strokes. Then, building upon this renderer, we design algorithms to automatically create 3D models with geometric details from a single‐view sketch. With the aid of two introduced loss functions: one based on silhouette‐derived confidence maps and the other on regression similarities, our framework supports the gradient of loss functions calculated between the rendered sketch and input sketch back‐propagating through the whole architecture, thereby enhancing the geometric details on the frontal surface of the generated 3D object. Through comparisons with state‐of‐the‐art sketch‐based 3D modelling techniques, our approach demonstrates superior capability in generating plausible geometric shapes and details, without the necessity for semantic annotations within the input sketch.