Memory-Augmented Transformer for Relation Extraction of entities

This work proposes a model for NLP tasks that extends a Universal Transformer (UT), a parallel-in time self-attentive recurrent sequence model, to take advantage of the dynamic external memory controller of a Differentiable Neural Controller (DNC). The UT has high intrinsic parallelism and uses dynamic halting to control the number of computational steps needed to process each input of the sequence. The proposed model improves the UT by adding a char-n-gram-based word representation in the input encoder and proposes a linear bottleneck residual block with batch normalization as a transition function, thus improving the feature learning inside the Transformer. Furthermore, this work modifies the original memory access controller of the DNC to process parallel-in time recurrent sequences and shows significant speedup over the original. Performance evaluation considers three benchmarks datasets for a relation extraction task. The paper also shows that the proposed model outperforms previous neural network approaches.