Learning to Rank Onset-Occurring-Offset Representations for Micro-Expression Recognition

This paper focuses on the research of micro-expression recognition (MER) and proposes a flexible and reliable deep learning method called learning to rank onset-occurring-offset representations (LTR3O). The LTR3O method introduces a dynamic and reduced-size sequence structure known as 3O, which consists of onset, occurring, and offset frames, for representing micro-expressions (MEs). This structure facilitates the subsequent learning of ME-discriminative features. A noteworthy advantage of the 3O structure is its flexibility, as the occurring frame is randomly extracted from the original ME sequence without the need for accurate frame spotting methods. Based on the 3O structures, LTR3O generates multiple 3O representation candidates for each ME sample and incorporates well-designed modules based on learning to rank (LTR) to measure and calibrate their emotional expressiveness. This calibration process implicitly enhances the visibility of MEs by amplifying the originally narrow emotional expressiveness gap among ME frames caused by their low-intensity characteristics, thereby facilitating the reliable learning of more discriminative features for MER. Extensive experiments were conducted to evaluate the performance of LTR3O using four widely-used ME databases: CASME II, SMIC, SAMM, and MEVIEW. The experimental results demonstrate the effectiveness and superior performance of LTR3O, particularly in terms of its flexibility and reliability, when compared to recent state-of-the-art MER methods.