Evaluation of Sense Embeddings with Optimized Vector-Meaning Correspondence

A word sense is an essential element for understanding what a sentence means and can be interpreted as a concept on its own. To realize this cognition in Computational Linguistics, embedding methods have been proposed to map words to dense vectors. Among them, sense embeddings assign multiple vectors to each word to represent its distinct meanings. Their special feature is that the boundary between the meanings of each word explicitly exists. However, their qualities are evaluated using a conventional approach to word embeddings that implicitly addresses meaning. In precise, these evaluations adopt datasets composed from combinations of pairs of words and similarities between two words, where the number of meanings to be evaluated is limited compared to the number of words. Moreover, their evaluation metrics reflect only a part of the relationships between multi-sense words. To overcome these problems, in this paper, we propose a novel evaluation method to sense embeddings that covers rich meanings and addresses the combinations arising from polysemy, such as the uniqueness and redundancy of vectors. Our key idea is a vector, appropriately representing its meanings, has neighbors that can be considered to be similar words in a vector space. Based on this idea, we automatically construct an evaluation dataset with similar words for each meaning by combining information from two reliable concept hierarchies; one is manually managed, and the other is automatically created and manually managed. Then, based on the constructed dataset, we devise three kinds of evaluation metrics that associate vectors of a multi-sense word with its meanings in the dataset in different manners. Through an experiment, we empirically show that the proposed evaluation method can adequately reflect the quality of sense embeddings compared to the conventional method.