Software-defined event-based camera and vision algorithms for digitizing highly dynamic manufacturing tasks

Event-based cameras are inspired by the way the retina in the eye process information. Rather than capturing image frames at a constant rate, event cameras generate data asynchronously, based on changes in pixel value between frames. As a result, these cameras have a high spatiotemporal accuracy, low power constraints and reduced artefacts from motion blur. Hence, they represent favourable properties for digitizing highly dynamic manufacturing tasks such as machine vibrations and rapid human assembly actions. However, the costs of event-based cameras are prohibitively high for small to medium sized enterprises (SMEs). In this research, we present the development of a rig that combines low-cost visual sensors with low-cost algorithms to perform action recognition for highly dynamic manual manufacturing tasks. We applied our algorithms to achieve an action classification system via a novel threshold-based approach and in parallel use a neural network to perform classification of objects in the video feed. The results achieved demonstrated that our approach was able to successfully identify various rapid manual tasks being performed on identified objects. In addition to SMEs, our approach would be suited for providing low-cost digitization on legacy machines.