Estimating Story Points in Scrum: Balancing Accuracy and Interpretability with Explainable AI

Accurate story point estimation is crucial for agile software development. Manual estimation is time-consuming and error-prone, leading to project delays and budget overruns. This study introduces a novel data preprocessing approach to address these challenges. Our method combines similar user story removal with conflicting story point values and data augmentation, distinguishing it from prior research. We contribute 3 new public datasets to support this research. Our experiments demonstrate the proposed preprocessing significantly enhances model accuracy, with the Support Vector Machine (SVM) achieving the highest performance (43.21%) compared to Bidirectional encoder representations from transformers (BERT) (42.37%). We use Local Interpretable Model-Agnostic Explanations (LIME), Shapley Additive Explanations (SHAP), and Transformer-Interpret to enhance model interpretability. Among the explored interpretability approaches, highlighting relevant words proved the most effective for understanding model behavior. A human-centered evaluation assessed the model performance and reliability of the explainability methods. While both SVM and BERT had limitations in replicating human estimations, SVM performed marginally better. SHAP outperformed LIME and Transformer-Interpret in aligning with human reasoning, effectively capturing salient data aspects and showing promise for enhancing human-AI collaboration in agile development. Our findings contribute to advancing story point estimation accuracy and transparency in agile software development.