Wednesday, December 9, 2009

How to make computers that work like the brain

This is the interesting title of a short article presented by Dileep George (co-founder of Numenta) back in July 2009. It is available at the following link, but you need access to an educational database (such as through a University) to get to it:

A couple of interesting points were made in the article: As was mentioned publicly by Subutai Ahmad in September, Numenta's third generation of algorithms are now being developed. This means that the algorithms are beginning to become more mature. As a result, Numenta is finally starting to think seriously about implementing HTM in hardware. Until now, HTM was such a new technology, and the algorithms have been so early in their development, that it was not feasible to use hardware.

It turns out that HTM in its software configuration is very CPU hungry, so a hardware implementation could be quite useful. For instance, Vitamin D's new beta webcam software that can recognize people in video can only analyze two 320 by 240 video feeds for each dual core 2 GHZ processor. Think about that. That means that if Vitamin D were to use its HTM-based software to analyze 640 by 480 video, it would require a quad core 2 ghz processor for EACH webcam feed. Extrapolating from that, it would be difficult to find a home computer powerful enough to analyze a single 1080P or even a single 720P video feed.

Perhaps software versions of HTM would be sped up significantly with GPU computing, however. HTM can apparently scale to a very large number of parallel CPU's, so one would think that a GPU could improve significantly over a CPU's performance.

Sunday, November 29, 2009

Turning point for HTM- Vitamin D releases first commercial HTM app.

Earlier this month, Vitamin D, Inc. issued a press release noting that it released a beta of video surveillance software based on Numenta's hierarchical temporal memory. This is the first commercial application of HTM, which represents a significant milestone for the technology. The software has been trained to recognize the presence of people in videos, which marks a significant improvement over less intelligent technologies that simply look for motion in video. It sounds as though future versions of the software will be able to distinguish different actions taken by people, which should open up many further commercial HTM possibilities. This is only the beginning for commercial HTM apps.

Friday, November 27, 2009

Just posted a link to a September 2009 speech regarding HTM

This is the most recent speech I can find by a Numenta employee regarding the HTM technology. In it, Subutai Ahmad goes through the basics of Hawkins' theory, and outlines how it is currently being implemented in software. A couple of interesting tidbits from it: Ahmad mentioned that Numenta is currently creating its third generation of algorithms. He also mentioned that Numenta plans in 2010 to release a Prediction toolkit (on the heels of its recent release of the Vision toolkit). The vision toolkit allows individuals without any programming skill to create a working computer vision system that can recognize objects in grayscale images. The second generation of algorithms are what allowed the technology to advance to allow such an application. I anticipate that the release of the Prediction toolkit will accelerate the race toward many commercial applications of the HTM technology.

As anyone who read Hawkins' book knows, he argues in it that prediction is the essence of higher intelligence. The ability to predict the future by way of the human neocortex is what separates us from animals such as dogs and cats. The heart of the theory is that columns of neurons in our brains are continuously looking for patterns in the sensory data presented to them. Patterns that follow each other in time (sequences of patterns) are stored for later use. Predictive abilities rise in the brain due to its ability to recognize that it is in a sequence that it has seen before (i.e. when you are walking down the hall of your home, your brain predicts that you will shortly enter your kitchen). The second generation of HTM algorithms only implemented predictive capabilities in a very, very limited way. Presumably, the third generation of algorithms will allow for broad predictive capabilities in Numenta's software. Ahmad demonstrated one interesting prediction application using HTM. Working with a well known financial publication, Numenta stored thousands of sequences in its HTM hierarchy showing the click patterns for users of the publication's website. The idea was to see if the HTM software could predict what topic a user would click on next, given the history of which articles the user had previously clicked on. While random chance would provide less than 1% accuracy (there were nearly 200 topics to choose from), the software predicted with 45% accuracy which topics users would click on next. This is but one example of how HTM will allow for tremendously useful web traffic analysis.

Why a blog about HTM?

Of all the topics that one could write about, why write about a fairly obscure new machine learning technology known as hierarchical temporal memory (HTM)? The fact is, I have been interested in the idea of intelligent machines since I was a kid back in the 1980's. The idea of robots and computers that could think for themselves was fascinating to me. At the time, it seemed as though the idea of artificial intelligence was dead. I lost interest in technology for a very long time (roughly ten years from the early 90's until late 2004). One night, with nothing better to do, I went on the internet, and went looking for tech websites. I found that times had changed, somewhat. Robots were still lumbering zombies, but it still seemed as though significant progress was being made with getting computers to think for themselves.

At that time, I also heard about a new book by Jeff Hawkins (famous for the creation of the Palmpilot and Treo) called "On Intelligence." I did not read the book at the time (I was in the middle of law school with a wife and new baby so pleasure reading was not high on my list at that time). Yet, I heard that the book, a theory of brain intelligence and how computers could be built on similar principles, was well received even by many neuroscientists. I wondered if, perhaps, Mr. Hawkins was on to something better than the many failed AI experiments of the past. Hawkins himself alluded in his book to many of these failures dating back to the beginning of the computer age. About a year ago, I saw someone on a website mention Numenta, Hawkins' new company that was created to replicate in computers his brain theory set forth in "On Intelligence." This individual essentially said that Numenta had failed to deliver on Hawkins' promises. I decided to go back and read "On Intelligence."

I now have a wife and three small children, but my daily train commute allows for plenty of pleasure reading. Early this year, I finally sat down and read the book, and was blown away by it. In my mind, Hawkins brilliantly explained why the field of artificial intelligence has largely failed in its goal of implementing any degree of general intelligence in computers. Further, he made a plausible case that it wouldn't be terribly difficult to begin to implement his theory into hardware and software. I wondered whether the person mentioned above was correct in his assessment that Numenta was largely a failure. I found quite the contrary. Numenta has made steady progress since its founding in early 2005 toward its ultimate goal of creating a brain-like computer. I will try to discuss some of this progress here in this blog.

I want to make a disclaimer. I am not an engineer, computer scientist, or neuroscientist by trade, so this blog will definitely be less technical than what you might see from other tech blogs. While I am interested at some level in the nuts and bolts of Numenta's technology, my knowledge certainly does not extend to the mathematical theory behind HTM.

My blog is meant simply to be a repository for any news about HTM and Numenta. There are many interesting news articles and videos on the topic dating back several years, but they are spread across the entire web. Numenta's website is a good start, but there is so much more out there.