Quantized large language models in biomedical natural language processing: evaluation and recommendation

Abstract Large language models have demonstrated strong capabilities in biomedical natural language processing, yet their rapid growth in size and computational requirements creates a barrier to local deployment in privacy-sensitive settings. In this study, we evaluated the impact of quantization on 12 general-purpose and biomedical large language models across eight benchmark datasets covering named entity recognition, relation extraction, multi-label classification, and question answering. Using Transformers with BitsAndBytes, we found that 4-bit and 8-bit weight quantization substantially reduced peak GPU memory while often preserving task performance, although latency generally increased and degradation was task- and model-dependent. We further evaluated deployment-oriented AWQ, GPTQ, and FP8 checkpoints with vLLM on representative biomedical tasks and model scales. We also quantified invalid-output rates, prompt overhead, and few-shot memory effects. These results provide a systematic benchmark of quantization trade-offs for biomedical NLP and show that weight-memory reduction can make otherwise difficult-to-host large models feasible for local deployment on GPUs with sufficient VRAM, including high-memory consumer GPUs.