Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs

In this work, we explore how the inference time of a Transformer Neural Network can be efficiently optimized with applications to real-time anomaly detection in financial time series. The financial time series are price series such as asset prices. Unfortunately, the data are often with errors or outliers that make the downstream data processing tasks useless, unstable or even harmful [Falkenberry, T. N., 2008, High Frequency Data Filtering, available at: https://www.tickdata.com/whitepaper/high-frequency-data-filtering ; Vallis, O., J. Hochenbaum and Twitter, 2015, Introducing Practical and Robust Anomaly Detection in a Time Series, available at: https://blog.twitter.com/engineering/en_us/a/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series ]. Moreover, the amount of financial time-series data has been significantly increasing [Ahmed, M., N. Choudhury and S. Uddin, 2017, Anomaly Detection on Big Data in Financial Markets, in 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), IEEE, pp. 998–1001]. Hence, there is a need for better data-cleaning methods in terms of accuracy and in terms of processing speed. Transformers as a neural network architecture have achieved superior performances in many tasks such as Natural Language Processing and Computer Vision [Gillioz, A., J. Casas, E. Mugellini and O. A. Khaled, 2020, Overview of the Transformer-based Models for NLP Tasks, in 2020 15th Conference on Computer Science and Information Systems (FedCSIS), pp. 179–183]. Time series modeling and especially anomaly detection tasks can benefit from the features of transformers architecture in multiple ways, including the capacity to capture long-range dependencies and interactions [Wen, Q., T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan and L. Sun, 2022, Transformers in Time Series: A Survey]. Increasingly powerful hardware, such as field-programmable gate arrays (FPGAs), have seen increasing usage in recent years due to their reconfigurability and high performance [Funie, A.-I., L. Guo, X. Niu, W. Luk and M. Salmon, 2017, Custom Framework for Run-time Trading Strategies, in S. Wong, A. C. Beck, K. Bertels, and L. Carro, editors, Applied Reconfigurable Computing, Springer International Publishing, Cham, pp. 154–167]. They can be efficiently utilized to speed up the computations of the Transformer architecture. We explore different Transformer architectures for time series modeling and how they can be efficiently implemented on an FPGA board (PYNQ-Z2). In particular, we examine the application of Transformers to detect anomalies in time series and we show how they can be efficiently implemented on an FPGA board to minimize latency. The code is available at https://github.com/thxi/icl_thesis .