Towards Developing a Comprehensive Methodical Framework for Evaluating Code Recommender Systems

The growing incorporation of code recommender systems is transforming software development, promising increased productivity, efficiency, and effectiveness through context-aware code suggestions.Despite these advancements, existing evaluation methods often fall short.They rely on computational metrics and synthetic benchmarks, failing to capture realworld software development environments' dynamic, human-centric nature.To enhance methods, techniques, and tools for evaluating code recommender systems, the aim is to create a new methodological framework.To achieve this goal, a multi-method approach will be employed, including a systematic literature review, along with user studies and online evaluations conducted in both a university setting and the automotive sector.By addressing practical challenges and involving human participants, the framework aims to thoroughly assess code recommender systems, ultimately providing empirical evidence of their impact on developers' productivity, efficiency, effectiveness, and perceived satisfaction.Furthermore, the research offers practical recommendations for designing more effective code recommender systems based on developers' needs and experiences.