Locally-deployed Open-source LLMs for Code Generation: Promises and Challenges

This study examines the feasibility of deploying open-source large language models (LLMs) locally for code generation, addressing key concerns related to security, cost efficiency, and operational autonomy. We conduct a rigorous evaluation of six open-source LLMs using the HumanEval benchmark, analyzing their performance across five prompting techniques: Zero-shot, Test-driven, Chain-of-Thought, MapCoder, and LLMDe-bugger. Our results indicate that code-specialized models, when combined with advanced debugging methods, achieve pass@1 accuracy rates of up to 88.02%, rivaling cloud-based solutions while ensuring data sovereignty. Moreover, we explore the trade-offs between model specialization, prompting complexity, and computational overhead, offering practical guidance for organizations considering local LLMs deployment. Our findings affirm that well-configured local LLMs can provide secure and cost-effective alternatives for sensitive software development workflows without compromising performance.