Today's post will be about sparseness.
The basic idea is that, if you look in my brain while it's processing an image, there will only be a small number of nerve cells active at any time. So, while the input image comes in as millions of numbers (the activity values of all the photoreceptors on my retina), my visual cortex is representing that image in terms of a much smaller number of variables.
This is good for a lot of reasons: it reduces the amount of energy I need to spend on image processing (small number of active neurons means less energy, and my brain takes up a lot of my body's energy budget), reduces the number of values that need to be passed on to the next stage of sensory processing, and it makes the input "simpler".
What do I mean by simpler? Well, on some level, my brain is seeking to "explain" the input image, in terms of a (usually small) number of relevant "causes". As an example, my desk right now contains a laptop, a coffee cup, and a picture of my girlfriend. If I want to make behavioral decisions, that's probably enough information for me: I don't need to actively consider all of the messy details of each of those objects, although I can figure them out of I want to.
So, by maintaining a sparse representation, my brain is forcing itself to find the relevant information, while filtering away a lot of the unnecessary details. For this reason, sparseness is one of the most important ideas in all of unsupervised learning.
Indeed, almost every paper published in the last 15 years about coding of sensory inputs boils down to seeking sparse representations of naturalistic stimuli.
The cool thing is that the guys who invented this notion work just down the hall for me. Berkeley FTW!
No comments:
Post a Comment