Fine-Tuning Locally Deployable Scientific Language Models: Reliability and Resource Tradeoffs

Commercial cloud-based language models increasingly support scientific research, but transmitting unpublished data raises concerns about privacy and institutional control. Small, locally deployable open-weight models offer an alternative, yet whether domain fine-tuning improves scientific reliability enough to justify its resource costs remains uncertain. Using electrochemistry as a test domain, we converted 4,983 open-access papers into 86,329 questionanswer pairs and fine-tuned eleven models (3-32 billion parameters) using low-rank adaptation. We compared adapted and original models on 8,481 held-out questions using reference-similarity metrics, three LLM judges and a blind five-expert pilot, and evaluated continued pretraining and retrieval-augmented generation (RAG) alongside financial and environmental costs. Fine-tuning increased ROUGE-1 reference-answer similarity for ten models (11.1-86.5%) but lowered judged answer quality in 18 of 19 configurations (mean change −0.621 on a five-point scale), consistent with expert preferences. Apparent gains depended on model, checkpoint, training seed, and answer length, while larger models or more synthetic data did not consistently help. Qualitative examples also exposed overconfident errors. Continued pretraining alone offered limited benefit; combining it with supervised fine-tuning improved scores over fine-tuning alone but not over unmodified models in judged quality. In a single-model, open-book pilot, RAG achieved higher judge scores at $6 versus $302 in development cost, with estimated energy, emissions, and water footprints of 2.8 kWh, 0.9 kg CO2e, and 6 L versus 132 kWh, 44 kg CO2e, and 271 L, despite 2.0-3.1-fold more computation per query. These results support retrieval with an unmodified local model as a practical baseline for privacy-sensitive research and highlight the need to justify finetuning through independently verified accuracy, robust benchmarking, and full resource costs rather than textual similarity alone.

Authors

Institutions

Publication Details

Journal
ChemRxiv
Published
2026-11-12
DOI
https://doi.org/10.26434/chemrxiv.15012130/v1
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Fine-Tuning Locally Deployable Scientific Language Models: Reliability and Resource Tradeoffs

Abdelrahman Refaie, Mohsen Afshari, Sheida Shakoorian, Mohammad Rahimi et al.
ChemRxiv
Topic Modeling
preprint

Fine-Tuning Locally Deployable Scientific Language Models: Reliability and Resource Tradeoffs

Abdelrahman Refaie, Mohsen Afshari, Sheida Shakoorian, Mohammad Rahimi, Prince Aleta, Ahmad Hassan
preprint en

Abstract

Commercial cloud-based language models increasingly support scientific research, but transmitting unpublished data raises concerns about privacy and institutional control. Small, locally deployable open-weight models offer an alternative, yet whether domain fine-tuning improves scientific reliability enough to justify its resource costs remains uncertain. Using electrochemistry as a test domain, we converted 4,983 open-access papers into 86,329 questionanswer pairs and fine-tuned eleven models (3-32 billion parameters) using low-rank adaptation. We compared adapted and original models on 8,481 held-out questions using reference-similarity metrics, three LLM judges and a blind five-expert pilot, and evaluated continued pretraining and retrieval-augmented generation (RAG) alongside financial and environmental costs. Fine-tuning increased ROUGE-1 reference-answer similarity for ten models (11.1-86.5%) but lowered judged answer quality in 18 of 19 configurations (mean change −0.621 on a five-point scale), consistent with expert preferences. Apparent gains depended on model, checkpoint, training seed, and answer length, while larger models or more synthetic data did not consistently help. Qualitative examples also exposed overconfident errors. Continued pretraining alone offered limited benefit; combining it with supervised fine-tuning improved scores over fine-tuning alone but not over unmodified models in judged quality. In a single-model, open-book pilot, RAG achieved higher judge scores at $6 versus $302 in development cost, with estimated energy, emissions, and water footprints of 2.8 kWh, 0.9 kg CO2e, and 6 L versus 132 kWh, 44 kg CO2e, and 271 L, despite 2.0-3.1-fold more computation per query. These results support retrieval with an unmodified local model as a practical baseline for privacy-sensitive research and highlight the need to justify finetuning through independently verified accuracy, robust benchmarking, and full resource costs rather than textual similarity alone.

ChemRxiv
Harvard University (US), University of Houston (US)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon