EFFECTIVE MAE PRE-TRAINING OF TRANSFORMERS FOR EMOTION RECOGNITION

The paper is dedicated to studying the pretraining of transformers on large volumes of face images using CrossMAE and finetuning for facial emotion recognition (FER). It provides a short summary of the current state of the problem as well as of it’s solutions. The study demonstrates how pretraining visual transformers with MAE employing crossattention in the decoder, combined with the Lion optimizer, flashattention 2.0, and other elements, provides a robust foundation for finetuning for FER.