Converge: towards an efficient multi-modal sensing research infrastructure for next-generation 6 G networks

Telecommunications and computer vision solutions have evolved significantly in recent years, allowing a huge advance in the functionalities and applications offered. However, these two fields have been making their way as separate areas, not exploring the potential benefits of merging the innovations brought from each of them. In challenging environments, for example, combining radio sensing and computer vision can strongly contribute to solving problems such as those introduced by obstructions or limited lighting. Machine learning algorithms, able to fuse heterogeneous and multi-modal data, are also a key element for understanding and inferring additional knowledge from raw and low-level data, able to create a new abstracting level that can significantly enhance many applications. This paper introduces the CONVERGE vision-radio concept, a new paradigm that explores the benefits of integrating two fields of knowledge towards the vision of “View-to-Communicate, Communicate-to-View”. The main concepts behind this vision, including supporting use cases and the proposed architecture, are presented. CONVERGE introduces a set of tools integrating wireless communications and computer vision to create a novel experimental infrastructure that will provide open datasets to the scientific community of both experimental and simulated data, enabling new research addressing various 6 G verticals, including telecommunications, automotive, manufacturing, media, and health.