Do We Need to Ask Our Users? Probing the Efficacy of User Behavior Analytics

The increasing popularity of analytics-driven design questions the role of direct user feedback to improve User Experience (UX). In this study, we first deployed an interaction tracking-tool on existing software to inform design decisions in an analytics-driven prototype. Second, we collected direct user feedback on this version and iterated the design, resulting in the feedback-prototype. A meta-evaluation study comparing the UX of both prototypes and the existing software revealed that, the analytics-driven prototype was rated significantly higher on the UEQ-S questionnaire than the original software. Surprisingly, there were no significant improvements in perceived UX between the analytics-prototype and the feedback-prototype despite a reduction of required actions. We conclude, that UX can be significantly enhanced through behavior analytics alone. While differences between the two prototypes were little, relevant notions can only be retrieved directly from users. Future work needs to replicate these findings with independently developed, more diverse prototypes.