Assessing Usability of AutoML Tools: A Multi-Method Study

The Automated Machine Learning (AutoML) tools are introduced to make machine learning accessible to all users, regardless of their level of expertise and programming skills for analyzing variety of data, decision making, prediction etc. Such tools thus need to be usable for all type of users for maximum utilization of the tools. Although most of the studies are conducted on exploring effectiveness and efficiency of ML algorithms using the tools to predict, classify, diagnosis or taking decision, but a very few studies focused to enhancing its ease of use to make it accessible for all kind of users. While the ease of use or usability is one of the vital issues for the tool to be adopted by all kind of users from tech person to non-tech person. Therefore, this study aimed to evaluate the usability of AutoML tools. To attain this objective, we selected three AutoML tools Akkio, H20, and Amazon SageMaker Canvas based of predefined selection criteria. Then we evaluated the usability of the tools using Heuristic evaluation and user study. Heuristic evaluation were carried out by four experts and user study were conducted on ten participants considering a set of task. The heuristic evaluation revealed that the most commonly violated heuristics in Akkio were User Control and Freedom (H3) as well as lacking of Help and Documentation (H10). On the other hand, the user study results showed quite the contrary, with Akkio having an average task success rate of 93%. The results of the user study and heuristic evaluation may appear to be opposites at first glance, but they provide pathways for further research into the system's usability while revealing both its strengths and limitations. The outcome of this research will greatly contribute to design and develop the AutoML tools that would be usable to users having different expertise for decision making and data analysis using ML.