A Hybrid Deep Learning Framework Combining Convolutional, Transformer, and GRU Models with Mixture of Experts for RNA Sequence Analysis

This work proposes a novel hybrid deep learning model that incorporates CNNs, Transformer models, and GRU layers, further extended with an MoE model, for advanced sequence analysis. The proposed model leverages the strengths of each component: CNNs capture local dependencies, Transformers model long-range relations, while GRUs process sequential data with temporal dependencies. Thus, this Mixture of Experts architecture improves the model's capacity in terms of complex structures due to the specialized process it makes for different attributes of the inputs. During training, the learning rate schedule is performed with a warm-up and a transition to the cosine annealing schedule towards the best convergence. On evaluation with a benchmark RNA dataset, the model achieves competitive MAE scores of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 1 4 9 9 6}$</tex> for the public dataset and 0.17437 for the private dataset. This underlines its robustness and the capability to generalize while dealing with complex sequence data. The foregoing results point to the model's usefulness in areas that require precise sequence analysis, such as bioinformatics and genetic research.