Fine-Tuning LLMs in Information Retrieval for Historical Swedish Court Records Using AI-generated Question-Answer Data

The aim of this thesis is to fine-tune BGE M3-Embedding for retrieval in historical Swedish court documents. Historical language differs from contemporary language, and therefore requires specially adapted NLP tools. Multi-lingual embedding models have proven efficient for low resource languages, and can be fine-tuned for new languages and domains through question-answer datasets. BGE M3-Embedding combines dense, lexical and multi-vector retrieval techniques. We create question-answer training data by letting GPT-4o mini generate questions to archival texts. Results show that BGE M3-Embedding was worsened by the fine-tuning – it performs on a level similar to BM25 and below Swedish and multi-lingual dense retrieval models. The original M3-Embedding model outperforms other sparse and dense retrieval methods on the historical court data over different document lengths. Some possible reasons for the worsening are un-ideal fine-tuning methods, a large amount of uncontrolled, synthetic data and inability of large language models to interpret historical legal text.