Lexicons versus Corpora for Measures of Semantic Distance

Measures of semantic distance (or, inversely, semantic relatedness) have many applications in Computational Linguistics. There are three basic approaches to measuring semantic distance: lexicon-based algorithms, corpus-based algorithms, and hybrids. In an otherwise excellent paper on lexicon-based measures, Budanitsky and Hirst criticize corpus-based measures. I discuss their criticisms here.

Budanistky and Hirst give three criticisms of corpus-based measures:

First, while semantic relatedness is inherently a relation on concepts, as we emphasized in section 1.1, distributional similarity is a (corpus-dependent) relation on words. In theory, of course, if one had a large-enough sense-tagged corpus, one could derive distributional similarities of word-senses. But in practice, apart from the lack of such corpora, distributional similarities are promoted exactly for applications such as various kinds of ambiguity resolution in which it is words rather than senses that are available.

The problem with this criticism is that Pantel and Lin have shown that an unsupervised clustering algorithm can automatically discover word senses from text, without the need for manual sense tags. That is, distributional (corpus-based) approaches can infer word senses (concepts) from patterns of word usage.

Second, whereas semantic relatedness is symmetric, distributional similarity is a potentially asymmetrical relationship. If distributional similarity is conceived of as substitutability, as Weeds and Weir (2005) and Lee (1999) emphasize, then asymmetries arise when one word appears in a subset of the contexts in which the other appears; for example, the adjectives that typically modify apple are a subset of those that modify fruit, so fruit substitutes for apple better than apple substitutes for fruit. While some distributional similarity measures, such as cosine, are symmetric, many, such as alpha-skew divergence and the co-occurrence retrieval models developed by Weeds and Weir, are not.

There are at least three problems with this criticism: (1) The popular cosine similarity measure escapes this criticism. (2) If a function f(x,y) is asymmetrical, we can easily define a new symmetrical function g(x,y) as the average of f(x,y) and f(y,x). (3) There is evidence from cognitive psychology that human judgments of similarity are asymmetrical.

Third, lexical semantic relatedness depends on a pre-defined lexicographic or other knowledge resource, whereas distributional similarity is relative to a corpus. In each case, matching the measures to the resource is a research problem in itself, as this paper and (2003) show, and anomalies can arise. But the knowledge source for semantic relatedness is created by humans and may be presumed to be (in a weak sense) “true, unbiased, and complete”. A corpus, on the other hand, is not.

Bias in a corpus can be minimized by gathering a large amount of text; for example, a large collection of web pages. A manually generated lexicon, such as WordNet, will never match the coverage of Google. There is much bias involved in choosing which words and which word senses should be included in WordNet. There is much less bias in the variety of text indexed by Google.

We now turn to the hypothesis that distributional similarity can usefully stand in for semantic relatedness in NLP applications such as malapropism detection. Weeds (2003) considered the hypothesis in detail. She carried out a number of experiments using data gathered from the British National Corpus on the distribution of a set of 2000 nouns with respect to the verbs of which they were direct objects, comparing a large number of proposed measures of distributional similarity. … She found performance to be “generally fairly poor” (page 162), and undermined by the effects of varying word frequencies.

The British National Corpus consists of about 100 million words. This is very small by today’s standards. In my recent work, I have used a corpus of about 50,000,000,000 words, 500 times the size of the BNC.

A popular test for evaluating measures of semantic distance is a collection of 80 multiple-choice synonym questions from the Test of English as a Foreign Language (TOEFL). The highest TOEFL score for a lexicon-based measure is 78.75% (Jarmasz and Szpakowicz, 2003). The highest score for a corpus-based measure is 92.50% (Rapp, 2003). The highest score for a hybrid measure is 97.50% (Turney et al., 2003). To me, this is much more persuasive than the arguments of Budanitsky and Hirst.