Automated question generation from job descriptions using large language models: an evaluation of role-fit and fairness

This paper introduces an automated framework for generating high-quality, role-specific interview questions directly from complex job descriptions using an ensemble of Large Language Models (LLMs). We leverage seven advanced LLMs (gemma2-9b-it,gemma-7b-it,llama-3.3-70b-versatile,llama-3.1-8b-instant,llama3-70b-8192,mixtral-8x7b-32768, andOpenAI-3.5) to produce a diverse pool of questions. Our approach integrates prompt engineering and debiasing strategies to ensure that the generated questions are contextually relevant, role-aligned, and free from harmful biases. We compare the LLMs’ outputs against a strong baseline (ChatGPT o1 Pro) and human domain experts’ assessments, employing a multi-faceted evaluation framework including relevance, clarity, specificity, and fairness metrics. Our results, validated on a curated set of job descriptions across multiple industries, demonstrate that a multi-model generation strategy, combined with bias mitigation and expert-informed prompt design, yields superior interview question sets. This research offers insights into enhancing recruitment practices, improving efficiency, and fostering equitable candidate assessment.