Trans-ADENet: Transformer-based Attention-guided Deep Ensemble Network for high-dimensional data classification

High-dimensional (HD) biomedical data, such as gene expression profiles and ECG signals, pose significant challenges for machine learning (ML) due to limited sample size, feature redundancy, and noisy distributions. Conventional models tend to overfit, whereas boosting and ensemble approaches struggle with irrelevant features. Deep autoencoders (DAE) reduce nonlinear dimensionality but miss complex dependencies, whereas transformers require large datasets to model long-range relationships through self-attention mechanisms. We propose a Transformer-based Attention-guided Deep Ensemble Network (Trans-ADENet) that integrates dimensionality reduction, attention-driven feature learning, and meta-level ensemble fusion in an end-to-end framework. A deep autoencoder compresses HD inputs into compact latent representations, refined by a Transformer Encoder with multi-head self-attention. Refined features are fed to diverse base classifiers (CatBoost, Support Vector Machine (SVM), TabNet, and Generalized Multi-Layer Perceptron (GMLP)), and their outputs are fused by a meta-MLP, which learns adaptive weights to yield robust predictions. Experiments on breast, leukemia, INCART2 and Thyroid-RNA datasets achieved 96.3%, 94.1%, 92.7% and 94.6% accuracy, surpassing state-of-the-art models in terms of accuracy, F1, precision, recall, and AUC. By combining representation learning, attention, and adaptive fusion, Trans-ADENet delivers accurate, interpretable classification for biomedical tasks. • We design an integrated framework that combines deep autoencoding, transformer-based attention, and meta-level ensemble learning to tackle the challenges of high dimensionality, noise, and small sample sizes in biomedical datasets. • A deep autoencoder module is employed to compress noisy, redundant input features into compact, information-rich latent representations, thereby alleviating the curse of dimensionality and improving downstream learning efficiency. • The compressed latent features are further refined using a Transformer Encoder, which leverages multi-head self-attention to highlight globally informative and class-discriminative features dynamically. • We incorporate a diverse ensemble of classifiers (Cat-Boost, SVM, TabNet, GMLP) that capture complementary inductive biases, each tailored to distinct aspects of tabular, nonlinear, and structured data. • A lightweight Meta-MLP is introduced to perform meta-level fusion of the base classifiers’ outputs, learning dynamic, sample-specific weighting to enhance predictive robustness and generalisation.