Evaluating Embedding Models: Investigating Factors Affecting Performance Through the Medical Domain

In this paper, we investigated the performance of embedding models on two different domains, using models of different embedding dimensions, and even varying dimension to fully understand the impacting factors for the models. We ran tests for seven state-of-the-art models across natural questions and medial QA datasets to investigate the models' out-of-domain performance, and how factors such as the number of training parameters and fine-tuning affect accuracy and speed. We also explored the potential impact of several other variables, such as how model architecture could be the only factor affecting processing speed and how training data could have a significant effect on out-of-domain accuracy. The results show that the accuracy of the models follows a decreasing exponential relationship with the number of embedding dimensions and that the number of training parameters has little effect on model performance. The models we found to have the best accuracy were the nq-distillbert for natural questions and MedEmbed-base for medical QA. The model with the lowest encode time and the best accuracy-to-speed ratio by far across both datasets, as well as the best out-of-domain performance, was the all-miniLM.