Multi-Level 3D Facial Expression Editing Based on Multi-Branch Spatially Varying Convolutional Network

Aiming at the problems in 3D facial expression editing, such as the complicated operation of models, the low reality of generated expressions and the lack of detailed information, this paper proposes a multi-level 3D facial expression editing method based on multi-branch spatially varying convolutional network. The method can generate new expressions that are both realistic and rich in detail based on the displacement of control points on the facial model. Firstly, the facial mesh undergoes a high-frequency region segmentation module in the network model to identify the high-frequency regions. Then, the entire facial mesh and control point constraints are input to the coarse editing module to generate basic expressions. Simultaneously, the high-frequency regions and control point constraints are input to the fine editing module to generate rich expression details. Finally, the basic expressions and expression details are fused to obtain the new expression. This hierarchical processing allows the network to generate fine details at a relatively fast speed. By introducing spatially varying convolution, spatial features of irregular mesh can be better captured, thus improving the accuracy of expression editing. A spatiotemporal correlation criterion that combines vertex spatial proximity and motion consistency is proposed, and the K-Means clustering algorithm is improved to greatly enhance the rationality and accuracy of automatic high-frequency region segmentation. A new loss function combining vertex position and vertex normal constraints is constructed, effectively improving the overall accuracy of the network. Using the proposed methods, expression editing was performed on four different character models. Compared to other methods, the proposed method can generate highly realistic expressions with rich details according to the user’s editing requests.