Leveraging Large Language Models for Feature Selection in Drug Recommendation Systems

ABSTRACT Feature selection in clinical applications requires domain expertise to identify clinically meaningful predictors, presenting challenges for traditional statistical methods that struggle to incorporate semantic relationships and clinical reasoning. This study evaluates Large Language Model (LLM)‐based feature selection methods for drug recommendation tasks, investigating their potential to leverage pre‐trained clinical knowledge compared to traditional algorithmic approaches. We systematically evaluated six state‐of‐the‐art LLMs (GPT‐4, GPT‐3.5 Turbo, LLaMA variants, and DeepSeek R1) across multiple prompting strategies including rank‐based, score‐based, Chain‐of‐Thought reasoning, and a novel Tree‐of‐Thoughts for Feature Selection (ToT‐FS) framework. All methods were evaluated on the MIMIC‐III clinical database for drug recommendation, with downstream performance assessed using XGBoost classification. Performance was compared against traditional methods including MRMR, LASSO, and random selection baselines. Advanced LLMs achieved competitive performance with traditional methods, with macro‐F1 scores exceeding 0.95 through optimized prompting strategies, closely approaching MRMR (0.97) and LASSO (0.97) performance. Score‐based prompting significantly outperformed rank‐based approaches, improving GPT‐4 from 0.83 to 0.95 macro‐F1 score. The ToT‐FS framework demonstrated consistent high performance across different search depths. Notably, substantial performance disparities emerged between model generations, with older LLaMA 2 variants showing near‐random performance (F1 < 0.03), while advanced models demonstrated emergent clinical reasoning capabilities. LLM‐based feature selection represents a promising paradigm for clinical applications, offering competitive performance with traditional methods while providing enhanced interpretability and domain knowledge integration. The emergence of effective clinical feature selection capabilities at specific model scales suggests advanced LLMs have internalized substantial clinical knowledge, positioning them as valuable tools for clinical decision support systems requiring transparent, interpretable feature selection.