depth first search

Andrew Ng’s keynote at ICDL 2008 was quite enjoyable. If nothing else, he has killer demos. One of the themes of the keynote was a particular view that machine learning is moving towards solving learning problems with a paucity of available labels. He quoted Geoff Hinton’s work on deep belief nets as an example of a learning algorithm that learns through experience, as opposed to learning exclusively through labels. Hinton’s arguments is that there are more connections in the brain then could possibly be trained through labels alone.

As interesting as this trend is, it does beg the question: Where do labels come from? Now this isn’t an issue for many practical applications. The labels come from graduate students. But if we change the problem, look at it from the perspective of an AI agent, labels take on a somewhat mysterious quality.

Imagine you’re a robot waking up in the world. You’ve got a constant stream of sensory values at your disposal, a motor apparatus that you could use to interact with the world, and perhaps various primitive behaviors to get started learning who you are and what you can do.

This may seem like a contrived scenario, but I would argue this is precisely the scenario that all intelligent agents that we know of solve during the first years of life. It’s a scenario with some interesting properties. For one, even if labels are provided to the agent, how does the agent come to understand that the labels are, well, labels? These sorts of pernicious grounding problems abound.

So, if you’re like Geoff Hinton and you want to present your model as method of the way the brain really works, I think you’d better go through considerable effort to justify the use of labels anywhere in the learning process.

It turns out that optimizing warehouse tasks is hard.

As I near completion on my final project for a course on reinforcement learning, I came across the following from Sutton’s page on tile coding:

With the code described so far, there is a small probability that unrelated inputs will hash into some of the same tiles. In a group of tilings, usually there will be no more than one such “collision”, so that it is not a big problem; the learning process will sort it out. There will not be a big effect on performance unless the memory is too small or the hash functions are poorly designed. Nevertheless, the possibility of such a problem is annoying. When one’s program doesn’t work, there is a tendency, deserved or not, to suspect a failure of the hashing function.

I did, in fact, discover that my memory size was too small, resulting in a number of collisions. That was not the only problem with my agent, but one of many.

Of recent related significance, the UTCS/ART autonomous vehicle team did not make the finals in the Urban Challenge. One of the technical problems the team faced was a bad Ethernet cable that delayed critical sensor readings by as much as five seconds. The thread here is that debugging (in the classic sense as a programmer art) does not apply easily to systems that exhibit degrees of homeostasis or non-determinism (e.g. Ethernet protocol, TD-learning).