SIREN: Scalable isotropic recursive column multimodal neural architecture

While the allure of deep-learning-based multimodal architectures, with their ever-expanding parameter sets, may be undeniable, many practical scenarios demand a different approach and require highly adaptable, lightweight networks capable of operating efficiently on edge devices in real-world scenarios. This imperative shift arises from the fact that deep-learning-based frameworks face two prominent bottlenecks: (i) the confinement to a predefined range of modalities, often limited to video data, with limited adaptability, and (ii) their tendency to demand substantial memory resources, leading to slow processing time and a heavy reliance on extensive training datasets. In response to these challenges, we introduce “SIREN”, an innovative, scalable, lightweight, and high-speed recursive-column architecture purpose-built for both classification and regression tasks. Thanks to its meticulously designed block-based architecture, SIREN can effectively adjust its size and adapt to the complexity and the number of modalities present in the data and the limited size of datasets. SIREN commences with a modality-driven signal grouping strategy, followed by a 2D embedding that adeptly captures valuable temporal-frequency data from time series. Subsequently, patch embedding intelligently reduces internal resolution, significantly enhancing the efficiency of image processing. At the heart of SIREN reside recursive columnar architectures equipped with adaptable depth-point convolution blocks, scaled for complexity and quantity of modalities in the input data. To further enhance the processed outputs, we integrate a recurrent neural network to better model temporal dependencies. The effectiveness of SIREN is validated through empirical experiments conducted on six classification benchmark datasets. Additionally, we rigorously tested SIREN in regression tasks using our newly introduced dataset, “Pentostreda”. These comprehensive experiments firmly establish SIREN as a high-performing solution 1 1 Link to the code and dataset at https://github.com/hubtru/Siren . . • Introduces a multimodal isotropic architecture with < 18MB, < 4.5M params, 23ms/obs. • Scales to complex modalities via isotropic blocks, recurrent and column links. • Presents a 2D embedding that avoids compression, enabling flexible downsizing. • Achieves higher accuracy than SOTA, with recursion boosting semantic features. • Releases the Pentostreda dataset with five modalities for classification and regression.