A Heterogeneous FTJ-Based Computing-in-Memory Architecture for Vision Transformer Acceleration via Hardware and Algorithm Co-Design

Vision Transformers (ViT) have set new benchmarks in computer vision tasks but are often limited by their high computational and storage requirements. This work presents HEFA, a heterogeneous Computing-in-Memory architecture based on Ferroelectric Tunnel Junctions (FTJ), designed to accelerate ViT. By integrating an FTJ-based CIM Engine and a Digital Reconfigurable Engine with dynamic pruning, HEFA achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.36 \times \sim 35.7 \times$</tex> speedup and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.96 \times \sim 34.1 \times$</tex> energy efficiency improvements compared to GPU and state-of-the-art accelerators.