TabNet and TabTransformer: Novel Deep Learning Models for Chemical Toxicity Prediction in Comparison With Machine Learning

The prediction of chemical toxicity is crucial for applications in drug discovery, environmental safety, and regulatory assessments. This study aims to evaluate the performance of advanced deep learning architectures, TabNet and TabTransformer, in comparison to traditional machine learning methods, for predicting the toxicity of chemical compounds across 12 toxicological endpoints. The dataset consisted of 12,228 training and 3057 test samples, each characterized by 801 molecular descriptors representing chemical and structural features. Traditional machine learning models, including XGBoost, CatBoost, SVM, and a voting classifier, were paired with feature selection techniques such as principal component analysis (PCA), recursive feature elimination (RFE), and mutual information (MI). Advanced architectures, TabNet and TabTransformer, were trained directly on the full feature set without dimensionality reduction. Model performance was assessed using accuracy, F1-score, AUC-ROC, AUPR, and Matthews correlation coefficient (MCC), alongside SHAP analysis to interpret feature importance and enhance model transparency under class imbalance conditions. Cross-validation and test set evaluations ensured robust comparisons across all models and toxicological endpoints. TabNet and TabTransformer consistently outperformed traditional classifiers, achieving AUC-ROC values up to 96% for endpoints such as SR.ARE and SR.p53. TabTransformer showed the highest performance on complex labels, benefiting from self-attention mechanisms that captured intricate feature relationships, while TabNet achieved competitive outcomes with an efficient, dynamic feature selection. In addition to standard metrics, we reported AUPR and MCC to better evaluate model performance under class imbalance, with both models maintaining high scores across endpoints. Although traditional classifiers, particularly the voting classifier, performed well when combined with feature selection-achieving up to 94% AUC-ROC on SR.p53-they lagged behind the deep learning models in generalizability and feature interaction modeling. SHAP analysis further highlighted the interpretability of the proposed architectures by identifying influential descriptors such as VSAEstate6 and MoRSEE8. This study highlights the superiority of TabNet and TabTransformer in predicting chemical toxicity while ensuring interpretability through SHAP analysis. These models offer a promising alternative to traditional in vitro and in vivo approaches, paving the way for cost-effective and ethical toxicity assessments.