An Objective Effect Evaluation Framework for Vectorization Models on Patent Semantic Similarity Measurement

How to objectively evaluate the effect of different vectorization models in measuring similarity between patents is a fundamental issue, which can help to select high-performance vectorization models to support advanced patent services. Based on the rank consistency index and hypothesis testing approach, a framework for evaluating the effect of different vectorization models on patents’ similarity is proposed based on whether the model can accurately predict the similarity ranking of patents. Integrating the factors of time and technical field, an empirical study is conducted under the proposed framework to objectively evaluate the effect of six mainstream text vectorization models for assessing the semantic similarity of patents, which is evaluated based on Chinese patents (English Translation) from 2010 to 2024. The results show that the performance of Llama 2 is the best among six compared models in all years and in all technical fields. The proposed framework can objectively evaluate the similarity measurement effect of different vectorization models and provides a basis for the selection of the vectorization model for patent semantic similarity measurement for advanced patent services.