Effects of High-Order Co-occurrences on Word Semantic Similarity

Comments on:

Benoît Lemaire and Guy Denhière
Effects of High-Order Co-occurrences on Word Semantic Similarity

PMI-IR estimates the semantic similarity between a pair of words by how frequently they co-occur within a certain window of text. This simple measure of similarity is surprisingly good at recognizing synonyms: it seems that synonyms often appear close together in text. LSA goes beyond this simple measure by comparing the contexts in which the two words appear: synonyms tend to appear in similar contexts. Technically, we may say that PMI-IR uses first-order co-occurrence, whereas LSA includes higher-order co-occurrence information. PMI-IR is easier to compute than LSA, especially for very large collections of text, but PMI-IR ignores higher-order information. How useful is this higher-order information?

Lemaire and Denhière have performed an elegant experiment that answers this question. Their idea is to incrementally feed text into LSA and watch how the similarity measure evolves with each new paragraph of text. For a given pair of words, X and Y, each paragraph is assigned to one of five categories:

1. occurrence of X but not Y,
2. occurrence of Y but not X,
3. first-order co-occurrence (occurrence of both X and Y),
4. second-order co-occurrence (occurrence of neither X nor Y, but occurrence of at least three words, Z1, Z2, Z3, where each Zi has first-order co-occurrences with X and with Y in previously seen paragraphs),
5. three-or-more-order co-occurrence (anything not covered by the other four categories).

The similarity of X and Y is measured before and after the addition of each new paragraph, and the gain or loss in similarity is noted for each of the five categories. The gains and losses were averaged over 28 pairs of words as the number of paragraphs was incremented from 2,000 to 13,637. The average gain (loss) for each of the five categories follows:

1. occurrence of X but not Y: -0.15
2. occurrence of Y but not X: -0.19
3. first-order co-occurrence: +0.34
4. second-order co-occurrence: +0.05
5. three-or-more-order co-occurrence: +0.09

The results suggest that first-order co-occurrence is the most important contributor to determining similarity, but higher-order co-occurrence makes a significant contribution.

The authors conclude, “Our simulation shows that, although semantic similarity is largely associated to co-occurrence, which is coherent with the literature, the latter overestimates the former. High-order co-occurrences need to be taken into account. By only considering the frequency of co-occurrence as a measure of the semantic similarity, people probably introduce a bias.” This conclusion is slightly misleading. It is true that first-order co-occurrence (category 3) overestimates the semantic similarity (the sum of all five categories), but PMI-IR takes into account the cases where X appears without Y, and vice versa, so PMI-IR is best represented by the sum of the first three categories. Thus the results suggest that PMI-IR would tend to underestimate similarity as measured by LSA. Of course, the experiment is only concerned with LSA, not with PMI-IR, but it seems reasonable to assume that PMI-IR is approximated by the sum of the first three categories.

This paper presents a very interesting analysis of the role of different types of co-occurrence in LSA. It provides some new insight into how LSA works. The results suggest that LSA will outperform PMI-IR, when LSA is scaled up to the quantity of text that PMI-IR can handle.