In-sensor compressing via programmable optoelectronic sensors based on van der Waals heterostructures for intelligent machine vision
Efficiently capturing multidimensional signals containing spectral and temporal information is crucial for intelligent machine vision. Although in-sensor computing shows promise for efficient visual processing by reducing data transfer, its capability to compress temporal/spectral data is rarely reported. Here we demonstrate a programmable two-dimensional (2D) heterostructure-based optoelectronic sensor integrating sensing, memory, and computation for in-sensor data compression. Our 2D sensor captured and memorized/encoded optical signals, leading to in-device snapshot compression of dynamic videos and three-dimensional spectral data with a compression ratio of 8:1. The reconstruction quality, indicated by a peak signal-to-noise ratio value of 15.81 dB, is comparable to the 16.21 dB achieved through software. Meanwhile, the compressed action videos (in the form of 2D images) preserve all semantic information and can be accurately classified using in-sensor convolution without decompression, achieving accuracy on par with uncompressed videos (93.18% vs 83.43%). Our 2D optoelectronic sensors promote the development of efficient intelligent vision systems at the edge.
Authors
- Johnny Chung Yin Ho (ORCID: https://orcid.org/0000-0003-3000-8794)
- Yang Chai (ORCID: https://orcid.org/0000-0002-8943-0861)
- H. Wang (ORCID: https://orcid.org/0000-0001-8379-8238)
- Songcen Xu (ORCID: https://orcid.org/0000-0002-0022-0906)
- Chaoliang Tan (ORCID: https://orcid.org/0000-0003-1695-5285)
- Yunpeng Xia (ORCID: https://orcid.org/0000-0003-0777-2129)
- Jiajia Zha (ORCID: https://orcid.org/0000-0003-0551-285X)
- Hau Ping Chan (ORCID: https://orcid.org/0000-0003-0162-2031)
- Haoxin Huang (ORCID: https://orcid.org/0000-0001-8913-1768)
- Zhongrui Wang (ORCID: https://orcid.org/0000-0003-2264-0677)
- Ye Chen (ORCID: https://orcid.org/0000-0003-0821-7469)
- Wei Wang (ORCID: https://orcid.org/0000-0003-0945-5389)
- Yi Ren (ORCID: https://orcid.org/0000-0002-0231-8093)
- Peng Yang (ORCID: https://orcid.org/0000-0001-8775-585X)
- Long Zheng (ORCID: https://orcid.org/0009-0005-2134-8828)
- Shuhui Shi (ORCID: https://orcid.org/0000-0002-6319-4047)
- Yongji Wang (ORCID: https://orcid.org/0000-0001-7472-7489)
Institutions
- Hong Kong Polytechnic University (HK)
- City University of Hong Kong (HK)
- Chinese University of Hong Kong (HK)
- Shenzhen University (CN)
- Hong Kong University of Science and Technology (HK)
- Southern University of Science and Technology (CN)
- Shenzhen Technology University (CN)
- University of Hong Kong (HK)
Publication Details
- Journal
- Nature Communications
- Published
- 2025-04-24
- DOI
- https://doi.org/10.1038/s41467-025-59104-7
- Citations
- 33
- Primary Topic
- Advanced Memory and Neural Computing
- Type
- article
- Field-Weighted Citation Impact
- 24.10
Funders
- City University of Hong Kong
- National Natural Science Foundation of China
- Innovation and Technology Fund
- Young Scientists Fund