Programmer expertise classification using cognitive load and transfer learning

Abstract Context Evaluating a programmer’s performance by considering both technical and cognitive skills is indispensable in many software development tasks. Problem Technical skills are relatively easy to assess, whereas non-technical skills, such as long-term memory, critical thinking, and emotional intelligence, are often overlooked while measuring the performance of a programmer. Aim The primary objective of this work is to propose a reliable framework for classifying programmers into five framework-derived proficiency ranks. Methods Participants are given an assignment with both technical and non-technical questions. A cognitive load index that combines technical and non-technical skills is used to analyze performance and stratify it into five ranks. The proposed framework was evaluated using CatBoost and TabNet classifiers in a transfer-learning setup. It used the source dataset (Another researcher’s dataset) for pre-training and the proposed programmer dataset for fine-tuning. Extensive experiments, including confusion matrix analysis, 5-fold cross-validation, sensitivity analysis, and ablation studies, confirm the framework’s effectiveness and robustness. Results The fine-tuned CatBoost model outperformed the TabNet model, achieving an impressive 87% accuracy, a significant improvement over the pre-trained model’s 31%. The findings demonstrate that combining technical skills with non-technical aspects greatly enhances the accuracy of classifying programmer expertise. This offers a dependable method for assessing programmers in software engineering education. The study develops a robust framework for evaluating programming performance that incorporates both technical and non-technical skills, and it can be adapted for other cognitive and skill-based evaluations across different fields.