Design, Development, and Testing of a Smart Hand Tool: Achieving Work Task Recognition Using Synthetic Data and Edge Intelligence

Abstract This paper describes research toward developing smart hand tools that leverage artificial intelligence (AI) and sensors for use by human workers. Smart hand tools can provide useful feedback that can benefit human workers, contribute to worker training, and broaden participation in the skilled trade workforce. Specifically, the paper focuses on task recognition. Given the challenges of producing enough training data for machine learning (ML) using data purely from human-based testing, this paper shows how data synthetically-generated by a robot can be leveraged in the ML training process. The paper also demonstrates how fine-tuning ML models for individual physical tasks and workers can significantly scale up the benefits of using ML to provide this feedback. Experimental results show the effectiveness and scalability of this approach, including comparing test data size versus accuracy. In order for smart hand tools of the type introduced here to operate in real-time task recognition, as well as providing analytics on efficient and safe tool usage and operation, ML models need to be deployed ‘on tool’. This paper demonstrates how this can be accomplished by using a tinyML implementation. This paper provides a proof-of-concept for using automated platforms to help train smart tools, which will be essential given the wide range of uses for smart hand tools.