Exploration of Low-Energy Floating-Point Flash Attention Mechanism for 18nm FD-SOI CMOS Integration at the Edge

In this work we study the sizing and feasibility of a processing element for the calculation of the attention mechanism used in the Transformer models, using floating point representation. We show that the Flash Attention implementation is needed to enable circuit integration at the edge. Different floating point formats, both standard and non standard representations, were explored and synthesized in 18nm FD-SOI CMOS to compare their area and leakage power. The quality of the results is assessed against a pytorch BERT reference model. The best representation found is a non-standard 12-bit floating point architecture that computes a single block of Flash Attention. Once compared to a RISC- V CPU synthesized with the same technology, shows 10<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> less energy consumption and 10<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> less latency, whilst providing a result with low mean error.