Interruptibility of software developers and its prediction using psycho-physiological sensors: a replication

Abstract Background Interruptions at inopportune moments can negatively affect the perceived productivity of software developers. Previous research has attempted to predict the interruptibility of software developers using biophysical data to minimize interruptions at such moments. Aims First, we sought to replicate the results of a well-known study that predicted the interruptibility of software developers with high accuracy in lab and field settings. Second, we provide a comprehensive and publicly available lab package to facilitate future replications. Method We conducted an external close replication of the original lab study. As such, we held ten one-on-one sessions in which participants were asked to perform three code change tasks during a 60-minute programming session while frequently interrupted by a tablet application and monitored by selected biometric sensors. The biophysical data were used to derive predictive (classification) models. Results We cannot replicate models that significantly improve over the baseline of a majority classifier. Furthermore, we identify three severe threats to validity: a small sample size, a highly imbalanced dataset, and an ad hoc transformation of a critical measurement scale. Conclusions The original results did not generalize to our more varied setting (e.g., participant profile). We recommend only conceptually replicating the original study.