Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

Large Language Models (LLMs) perform differently on identical tasks when prompted in different languages—a phenomenon known as language bias. Although well-documented for general text generation, the extent to which language bias affects code generation quality and programming conventions remains largely unexplored. To address this gap, we investigate how the natural language used to describe programming tasks influences the quality of source code generated by three prominent LLMs: GPT-4o mini , DeepSeek, and Claude. Our study includes 460 coding tasks spanning Python (230 tasks) and Java (230 tasks). We translate and manually curate the original English prompts into four diverse languages: Chinese, Hindi, Spanish, and Italian, ensuring linguistic accuracy while preserving technical meaning. We evaluate the resulting code across multiple dimensions: functional correctness through test passage rates, structural quality via established code metrics, potential issues identified by static analysis tools, and lexical characteristics, including the natural language used in identifiers and comments. Results indicate that (i) source code generated from the English queries is not necessarily better in terms of passed test and quality metrics, (ii) the quality for different languages varies depending on the programming language and LLM being used, and (iii) the generated code tends to contain mixes of comments and literals written in English and the prompt language.