Enhancing Cancer Ontology-Based Query Retrieval with Transformer Models and Adaptive Angular Loss Functions

This paper investigates the enhancement of cancerrelated information retrieval using transformer models optimized with specialized angular loss functions. We compare BERT, RoBERTa, and GPT-2, trained with ArcFace and our novel FuzzyArcLoss [1], which dynamically adjusts margins based on confidence levels. Experiments conducted on NVIDIA DGX A100 GPUs reveal that FuzzyArcLoss significantly enhances retrieval performance for BERT (MRR: 1.0000, GPT-4 Score: 8.6 at <tex>$\tau=0.1$</tex>) and GPT-2 (MRR: 0.5000, GPT-4 Score: 7.0 at <tex>$\tau=0.5$</tex>), outperforming ArcFace in these contexts. RoBERTa with ArcFace achieves the highest overall performance (MRR: 1.0000, GPT4 Score: 7.0), though FuzzyArcLoss remains competitive. We recommend FuzzyArcLoss as the primary algorithm for BERT and GPT-2 due to its adaptability in ranking relevant ontology classes, with ArcFace optimal for RoBERTa. The study discusses the strengths of FuzzyArcLoss in handling variable data and proposes strategies to address GPT-2's limitations using newer autoregressive models. Practical steps for clinical implementation and experimental methodologies for future research are outlined.