Linear Decoding of Morphology Relations in Language Models (Student Abstract)

The recent success of transformer language models owes much to their conversational fluency, which includes linguistic and morphological proficiency. An affine Taylor approximation has been found to be a good approximation for transformer computations over certain factual and encyclopedic relations. We show that the truly linear approximation W s, where s is a early layer representation of the base form and W is a local model derivative, is necessary and sufficient to approximate morphological derivation, achieving above 80% top-1 accuracy across most morphological tasks in the Bigger Analogy Test Set. We argue that many morphological forms in transformer models are likely linearly encoded.