Freezing Scenarios for BERT Based Authorship Attribution

In this article pre-trained BERT model is fine-tuned for the authorship attribution of written articles. Different scenarios of the fine-tuning process, and structures of classification layers were tested on Reuter_50_50 dataset. Mainly the freezing/unfreezing scenarios of free parameters were designed and tested. These scenarios were dived into epoch based and layer based freezing strategies. To mitigate the effect of the attached classification layers, 2 models were tested, i.e. the simplest one (single linear layer) and a more complex one having 2 nonlinear layers. By these experiments it was shown that the classification performance is rather immune to the used classification module, i.e. the simplest one was comparable and even better that the more complex one. From the 2 tested freezing strategies the layer based was significantly better especially in terms of fine-tuning times. It was observed that the best performance (82.6%) is achieved when the last 8 layers of BERT are let unfrozen over the entire fine-tuning period.