Exact sequence interpolation with transformers

Abstract We prove that transformers can exactly interpolate datasets of finite input sequences in $$\mathbb {R}^d$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mrow> <mml:mi>R</mml:mi> </mml:mrow> <mml:mi>d</mml:mi> </mml:msup> </mml:math> , $$d\ge 2$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>d</mml:mi> <mml:mo>≥</mml:mo> <mml:mn>2</mml:mn> </mml:mrow> </mml:math> , with corresponding output sequences of smaller or equal length. Specifically, given N sequences of arbitrary but finite lengths in $$\mathbb {R}^d$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mrow> <mml:mi>R</mml:mi> </mml:mrow> <mml:mi>d</mml:mi> </mml:msup> </mml:math> and output sequences of lengths $$m^1, \dots , m^N \in \mathbb {N}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msup> <mml:mi>m</mml:mi> <mml:mn>1</mml:mn> </mml:msup> <mml:mo>,</mml:mo> <mml:mo>⋯</mml:mo> <mml:mo>,</mml:mo> <mml:msup> <mml:mi>m</mml:mi> <mml:mi>N</mml:mi> </mml:msup> <mml:mo>∈</mml:mo> <mml:mi>N</mml:mi> </mml:mrow> </mml:math> , we construct a transformer with $$\mathcal {O}(\sum _{j=1}^N m^j)$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>O</mml:mi> <mml:mo>(</mml:mo> <mml:msubsup> <mml:mo>∑</mml:mo> <mml:mrow> <mml:mi>j</mml:mi> <mml:mo>=</mml:mo> <mml:mn>1</mml:mn> </mml:mrow> <mml:mi>N</mml:mi> </mml:msubsup> <mml:msup> <mml:mi>m</mml:mi> <mml:mi>j</mml:mi> </mml:msup> <mml:mo>)</mml:mo> </mml:mrow> </mml:math> blocks and $$\smash {\mathcal {O}(d \sum _{j=1}^N m^j)}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mpadded> <mml:mi>O</mml:mi> <mml:mo>(</mml:mo> <mml:mi>d</mml:mi> <mml:msubsup> <mml:mo>∑</mml:mo> <mml:mrow> <mml:mi>j</mml:mi> <mml:mo>=</mml:mo> <mml:mn>1</mml:mn> </mml:mrow> <mml:mi>N</mml:mi> </mml:msubsup> <mml:msup> <mml:mi>m</mml:mi> <mml:mi>j</mml:mi> </mml:msup> <mml:mo>)</mml:mo> </mml:mpadded> </mml:math> parameters that exactly interpolates the dataset. Our construction provides complexity estimates that are independent of the input sequence length, by alternating feed-forward and self-attention layers and by capitalizing on the clustering effect inherent to the latter. Our novel constructive method also uses low-rank parameter matrices in the self-attention mechanism, a common feature of practical transformer implementations. These results are first established in the hardmax self-attention setting, where the geometric structure permits an explicit and quantitative analysis, and are then extended to the softmax setting. Finally, we demonstrate the applicability of our exact interpolation construction to learning problems, in particular by providing convergence guarantees to a global minimizer under regularized training strategies. Our analysis contributes to the theoretical understanding of transformer models, offering an explanation for their excellent performance in exact sequence-to-sequence interpolation tasks.