METNet: A mesh exploring approach for segmenting 3D textured urban scenes

In this work, we present the neural network Mesh Exploring Tensor Net (METNet) for the segmentation of 3D urban scenes, that operates directly on textured meshes. Since triangular meshes have a very irregular structure, many existing approaches change the input by sampling evenly distributed point clouds on the meshes. The resulting simplified representation of the urban scenes has the advantage that state-of-the-art neural network architectures for point clouds can be used for the segmentation task. The disadvantages are that crucial geodesic information is lost and for a sufficiently good approximation of the scene many points are often necessary. Since memory on the GPU is limited, the consequence is that only small areas can be examined locally. To overcome these limitations, METNet generates its input directly from the textured triangular mesh. It applies a new star-shaped exploration strategy, starting from a triangle on the mesh and expanding in various directions. This way, a vertex based regular local tensor from the unstructured mesh is generated, which we call a Mesh Exploring Tensor (MET). By expanding on the mesh, a MET maintains the information about the connectivity and distance of vertices along the surface of the mesh. It also effectively captures the characteristics of large regions. Our new architecture, METNet is optimized for processing METs as input. The regular structure of the input allows METNet the use of established convolutional neural networks. Experimental results, conducted on two urban textured mesh benchmarks, demonstrate that METNet surpasses the performance of previous state-of-the-art techniques. METNet improves the previous state-of-the-art results by 8.6% in terms of mean IoU (intersection over union) on the SUM dataset compared to the second-best method PSSNet, and by 3.2% in terms of mean F1 score on the H3D dataset compared to the second-best method ifp-SCN. Our source code is available at https://github.com/QenSchr/METNet .