Training Japanese Sentence-BERT with WikiLink Concept Base

In information processing, quantifying knowledge has always been crucial. Recently, vector databases that convert and store knowledge as vectors have become popular. However, these vectors must accurately reflect the original data’s meaning to be effective. This study focuses on converting sentences into vectors using machine learning models. For Japanese, existing datasets for machine learning lack a standard. To address this, we propose constructing a training dataset for the Japanese Sentence-BERT model using WikiLink Concept Base and Japanese Wikipedia. Models trained with this dataset outperformed comparison targets in various tasks using the XABC-STS test, demonstrating the effectiveness of our dataset and training approach.