Technical Document Query System using Transformer Model-based Machine Reading Comprehension

Constructing a Question Answering system is a challenging task despite a significant amount of study that has been conducted in recent times on this topic. It is even more difficult to provide satisfactory responses to the inquiries raised by users in an organizational setting as opposed to in an informal setting. We present in this paper, the results of our study into the use of a transformer-based model in the development of a technical document query system with machine reading comprehension. Our method fine-tunes a pre-trained transformer model with hyperparameter optimization using a pre-processed training dataset and tested on a different dataset. We experimented using eight pre-trained models from seven different variations of the BERT transformer architecture including BERT, RoBERTa, XLM-RoBERTa, ELECTRA, ALBERT, MobileBERT, and MPNet using the SQuAD1.1 dataset for fine-tuning and the Oracle Knowledge Documentation for testing. We found that the ALBERT pre-trained model is the best model achieving 0.891, 0.950, and 0.882 performance when measured using the Exact Match, F1 score, and Confidence Score metrics - despite its relatively small model size.