Designing vocabulary multiple-choice tests for exploring word frequency effects in distributional semantics

The representation of word meanings by positions in a vector space is now widely used across a range of psychological models and technological applications. There exist many different approaches for creating such vectors, with distributional semantic vectors (DSVs) often produced directly or indirectly from word co-occurrence statistics derived from large corpora of natural language. One of the most widely used methods for testing and comparing their performance is the vocabulary multiple-choice question (VMCQ) test, in which the best synonym of each tested word has to be identified from among a set of distractor words. By manipulating the details of the questions, they can also be used to explore finer-grained aspects of performance, in particular the effect of word frequency. Many such tests have already been created for testing human linguistic ability, and they are easily applied to DSVs by basing the choice on simple vector distances. This paper demonstrates that, while the general VMCQ approach is useful for testing DSVs, designing VMCQs specifically for DSVs provides better results than using existing tests created for humans. To do that, an improved VMCQ scoring method is first developed that provides considerably more statistically reliable results, and that allows a series of new evaluation methods to be introduced that enable improved explorations of the effect of word frequency on DSVs and VMCQs. These eventually lead to the identification of an improved general VMCQ approach for exploring frequency effects in DSVs, which is successfully tested on a representative set of nine previously published DSVs.