Adaptive learning behavior recognition using multimodal transformer based dynamic modality regulation

Abstract With multimodal interaction becoming increasingly common in intelligent learning systems, behavior streams from clicks, speech, facial expressions, and gestures often show time-varying reliability, short high-saliency events, and correlated or nested labels. These characteristics make static fusion and label-agnostic modeling unstable under imbalance and distribution shift. This paper proposes MFT-Net for an 18-label learning behavior recognition task in which multimodal sequences are mapped to behavior categories and optional nested sub-labels. MFT-Net integrates modal response filtering, dynamic modality weight regulation, and structured label embedding into a Transformer-based sequence model. The response filtering module preserves salient temporal segments while suppressing low-activity noise; the dynamic regulation module adjusts modality contributions according to cross-modal consistency; and the structured label embedding module introduces label semantics into attention-based sequence matching. Experiments on 5240 multimodal learning-behavior sequences show that MFT-Net achieves 96.2% accuracy and 95.5% F1 score, converges in 8 epochs, and obtains 34 ms average inference latency with an 18.6 MB model footprint under the workstation GPU setting. Robustness tests under skewed sample distributions and frame-length perturbations further indicate that the proposed model improves recognition stability while keeping computational cost within a deployable range.