Multimodal In‐Sensor Computing System Using Integrated Silicon Photonic Convolutional Processor

Photonic integrated circuits offer miniaturized solutions for multimodal spectroscopic sensory systems by leveraging the simultaneous interaction of light with temperature, chemicals, and biomolecules, among others. The multimodal spectroscopic sensory data is complex and has huge data volume with high redundancy, thus requiring high communication bandwidth associated with high communication power consumption to transfer the sensory data. To circumvent this high communication cost, the photonic sensor and processor are brought into intimacy and propose a photonic multimodal in-sensor computing system using an integrated silicon photonic convolutional processor. A microring resonator crossbar array is used as the photonic processor to implement convolutional operation with 5-bit accuracy, validated through image edge detection tasks. Further integrating the processor with a photonic spectroscopic sensor, the in situ processing of multimodal spectroscopic sensory data is demonstrated, achieving the classification of protein species of different types and concentrations at various temperatures. A classification accuracy of 97.58% across 45 different classes is achieved. The multimodal in-sensor computing system demonstrates the feasibility of integrating photonic processors and photonic sensors to enhance the data processing capability of photonic devices at the edge.