“The anchovies were nowhere near the sardines and the tuna. That’s because they were near the pizza toppings.
But it was only a problem because this was a three-dimensional grocery store. If it had been a thirty-dimensional grocery store they could have been near the pizza and the sardines.”
I’ve been putting off a post about phase space for nearly a year now, and watching this talk made me remember that I’ll soon have to do it.
Geoff Hinton talks here about a number of different spaces that are not the physical space that math was initially developed on.
- The bag of words model of a document takes each word in a text to be a dimension of the document, with like 80,000 possibilities each or however many words there are in English. (The 80,000 possibilities are the underlying corpus.)
- Latent semantic analysis of the bag-of-words type is how Google now does its search rankings. (PageRank only constitutes something like 30% of SERP ranking anymore, because [a] it’s too easy to game and [b] it’s inspecific to what’s being searched on. Domain authority and LSI comprise the rest. <—separate article)
- Seeing a pixel-by-pixel representation of a 2-D image as a list vector is problematic because the first pixel in a 200x300 Facebook profile image is next to the second pixel and also next to the 201st pixel. I.e. one needs a 2-array.
- Hinton talks about abstract feature space and energies — equivalently evolutionary fitness or economic utility — and ravines and mountains upon this manifold.
- The number of dimensions here is like the number of parameters (same as free parameters or degrees of freedom or arbitrary parameters in stats class) and in a neural net each “synapse” or graph edge is a lever you can pull.
- The same metaphor — and this is a metaphor in a grand sense which I hope to cover before the year is up — applies to the equalizer on your uncle’s home stereo, i.e. the number of terms in the Fourier decomposition.
