SVD, Variance, and Sparsity

There is a steady trickle of visitors to my post on Why Does SVD Improve Similarity Measurement?, so I gave this question a bit more thought. In that post, I offered three hypotheses about why SVD helps — high-order co-occurrence, latent meaning, and noise reduction — and I said that I didn’t know which hypothesis [...]

Code for Big Tensors

I recently completed a technical report on tensor decomposition, Empirical Evaluation of Four Tensor Decomposition Algorithms. The tech report has two messages: (1) If you like Singular Value Decomposition, you’re going to love Tucker Decomposition. (2) Many interesting applications involve large tensors. If your application requires large tensors, the tech report includes MATLAB source code [...]

SVD and Tucker Decomposition with Low RAM Requirements

Recently I’ve been experimenting with algorithms for the Singular Value Decomposition and the Tucker Decomposition, with the goal of processing large matrices (more than 105 rows and columns) and large tensors (more than 104 rows, columns, and tubes) that are relatively sparse (about 10% density). The problem with matrices and tensors of this size is [...]

Tensors for Data and Text Analysis

For the last several months, I’ve been playing with tensors as an approach to data and text analysis. Here are some pointers to get started on tensors.
Tensors are a generalization of matrices to higher dimensions:

order 0 tensor = scalar
order 1 tensor = vector
order 2 tensor = matrix
order n > 2 tensor = higher order tensor

PARAFAC [...]

Why Does SVD Improve Similarity Measurement?

In response to my earlier post on Effects of High-Order Co-occurrences on Word Semantic Similarity, Tom Landauer sent me the following note:
You have given me an idea. Because I have just been asked again to review papers that say that the way LSA works is by indirect associations, it seems that few have seen my [...]