Learning 2D Triangular Meshes from Images with Transformers

In this work, we present a method for learning 2D triangular meshes from images using the Transformer architecture.Creating polygonal representations of 2D surfaces from images has a wide range of applications, e.g.texture mapping and image segmentation.Current machine learning based methods either rely on deforming template meshes with fixed topology or require expensive post-processing to extract a planar polygonal mesh.In this paper, we demonstrate that deep learning can be utilized to directly generate a planar polygonal surface from arbitrary images without the need for additional input or constraints.Specifically, we show that the attention mechanism in transformer networks is highly effective in learning a unified representation of vertex positions and their neighborhood relationships.To showcase this, we propose a self-supervised training pipeline that enables end-to-end learning of meshes directly from images.We apply our approach to various datasets of handwritten letters and digits, and show that the model is capable of learning meshes with varying genus.