Analyzing and Interpreting Software Development Cost Estimation Error

Context: The degree of observed software development cost estimation error is sometimes used to make claims about software developers' estimation skills in empirical software engineering studies. The connection between estimation error and estimation skills may, however, be non-trivial. A careful analysis of other elements affecting the observed estimation errors may be needed to make valid claims about estimation skills and identify proper cost estimation improvements in empirical software engineering studies. Goal: This paper aims to model and analyze how observed estimation errors and estimation skills are connected to improve empirical software engineering research. Method: First, I present a model containing elements affecting the observed cost estimation error. Then, I discuss and illustrate what would be needed for additional information and analyses to derive estimation ability from the observed estimation error. Results: The estimation error-related elements included in the model are, in addition to estimation skill, the elements inherent (irreducible) estimation error, estimation error dynamics, lack of match between error measure and estimation type, selection bias, and measurement error of actual cost. The potential impact of these elements on the observed estimation error and how to analyze them are exemplified. Conclusion: It is challenging to derive estimation ability from the observed cost estimation errors. Despite this, it is important to be aware of how different elements not related to estimation skill affect the observed estimation error. This may not only help avoid a false attribution of high (low) estimation error to low (high) estimation skills but also as a means to examine the reasons behind the observed cost estimation error and identify proper improvement actions.