The Theoretical Foundation of Statistical Semantics

George Furnas defines Statistical Semantics as the study of “how the statistical patterns of human word usage can be used to figure out what people mean, at least to a level sufficient for information access”. Experiments show that statistical semantics works and it has many applications, but we may wonder why it works. How is it possible to figure out what words mean, simply by looking at patterns of words in huge collections of text? What is the theoretical foundation of statistical semantics?

I don’t have a complete and statisfying answer to this question, and we might not have an answer until we achieve true artificial intelligence. It’s tempting to say that the success of neural networks, machine learning, and statistical natural language processing is evidence that the human brain is fundamentally based on statistical algorithms, therefore it is not surprising that semantics should be based on statistics. However, this answer is only avoiding the question, by expanding its scope. Why do neural networks work?

In artificial intelligence, there is a debate between connectionism and computationalism. We may say that connectionism corresponds to statistical approaches to artificial intelligence and computationalism corresponds to logical approaches. It seems that statistical semantics must fall in the connectionist camp. There is a long tradition of looking at semantics from a logical perspective, and this leads to the question of how semantics can survive in the connectionist camp, without its traditional foundation in logic.

Peter Gärdenfors suggests a way out of this dilemma that I find appealing. In Conceptual Spaces: The Geometry of Thought, he presents a geometrical approach to semantics, as a kind of bridge between connectionist and symbolic approaches. I think this geometrical approach may provide a good foundation for statistical semantics. A geometrical approach is also offered in Geometry and Meaning and The Geometry of Information Retrieval.

Linear algebra, which is the basis for Latent Semantic Analysis, brings together all three of these themes: statistics, geometry, and logic. Geometry and Meaning shows how we can do Boolean logic with linear algebra operations. LSA is statistical in the sense that it can be used to find statistical patterns in data; it is geometrical in the sense that it is based on vectors, lines, and spaces; and it is logical in the sense that it supports Boolean operations. There are many gaps that need to be filled in here, but I can dimly see a theoretical foundation for statistical semantics.