Development of Resource Efficient Direct Speech to Speech Translation System using Transformer, Reformer, and Linformer

Direct speech to speech translation (DS2ST) refers to directly translating speech from one language into another language without relying on the written form. Developing such systems requires a significant amount of parallel data, which increases the demand for resources like GPU memory and training time, consequently raising costs. In this work, we focus on developing a DS2ST system using more memory and time-efficient transformer architectures such as Reformer and Linformer to address these challenges. A comparative analysis shows that Reformer accelerates training by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.4\times$</tex> and reduces the complexity from <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{O}(n^{2})$</tex> to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{O}(n\log n)$</tex> compared to the standard Transformer. Similarly, Linformer speeds up by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.7\times$</tex> and reduces the complex-ity of self-attention from <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{O}({n}^{2 })$</tex> to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathrm{O}(n)$</tex> in both time and space while maintaining the same level of translation accuracy of 16.46 BLEU score.