The Convergent Validity of Project Management Software Evaluation by Generative Artificial Intelligence Models

This article explores the convergent validity—or, more precisely, the inter-rater consistency—of popular generative artificial intelligence (AI) models in evaluating the quality of project management (PM) software. This study employed three prominent generative AI models—Gemini, CoPilot on Edge, and DeepSeek—to independently assign rating scores (1-10) to a curated list of 50 top-tier PM software systems/tools. The evaluation was structured around eight critical dimensions. Statistical analyses were conducted to assess the consistency of the AI-generated ratings. These included descriptive statistics to measure rating discrimination, analysis of mean absolute differences, paired-samples t -tests to identify erratic and systematic rating biases between AI model pairs, and Cronbach’s alpha to determine overall inter-rater consistency for each dimension. The results reveal a high degree of inter-rater consistency across all eight dimensions, a finding juxtaposed with the presence of significant systematic biases between the models. Cronbach’s alpha coefficients were found to be acceptable for all dimensions ( α = .782 to .897), indicating that the models consistently rate the PM software in a similar order. However, the t -tests confirmed that each model possessed a distinct "rating personality," consistently scoring higher or lower than its counterparts. These findings suggest that while generative AI is a surprisingly trustworthy measure for rating PM software roughly in a particular order, decision-makers must account for the distinct rating biases of each model. Relying on a single model for absolute rating scores is ill-advised, but leveraging multiple models to establish a consensus on relative quality is a viable and powerful new approach for PM software evaluation.