Machine Learning Jumpstart

The fallout from my Nature-Inspired AI book has been amazing, I could not be happier. Nevertheless, I've been pondering the idea of a follow-up book on Machine Learning. Something like: Clever Algorithms: Machine Learning Programming Recipes (Jan 2012). I've already started hacking technical reports on a private github repo (hopefully my wife doesn't read this blog).

Anyway, It has been ~10 years since I studied Machine Learning as an undergrad, and ~7 years or so since I hacked WEKA plugin's and fooled around with competition data like the KDD Cup. Sure, I've fooled around in competitions more recently, but I feel that I'm behind the state-of-the-art. Toward that end, I've been reading over what some of the top schools cover in modern Machine Learning courses and thought I'd capture my findings.

At this early stage, I'm just dumping the good online lectures, slides, and study notes. In no particular order:

• Stanford CS229, online lectures (also on itunes)
• MIT 6.867, lecture slides
• CMU 10-701/15-781, Machine Learning Department
• Berkley 294-fall09, Statistical Machine Learning Group
• Cornell COM S 478, CS 6780, ML Group

I'm very sure I've missed some excellent resources, please drop me a comment.

I'm finding a strong stats and Bayesian perspectives, which is a shift for me. I'm also finding topics like graphical models sneaking in. Lots of crossover in topics and syllabus, which is good. I would provide a condensed summary of topic areas here, but it looks really messy in my text file. I'd suggest that it would be better to just skim the lecture notes for each of the above courses to get the gist.

Historically I've found Data Mining books very useful given their process-centric practical focus. The top reference texts across these courses appear to be (linked on Amazon):

• Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
• Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.
• Duda, Hart and Stork, Pattern Classification, 2nd Edition, John Wiley & Sons, 2001.
• Mitchell, Machine Learning. McGraw-Hill, 1997.
• Sutton and Barto, Reinforcement Learning: An introduction, MIT Press, 1998.
• Hastie, Tibshirani, and Friedman. The Elements of Statistical Learning: Data Mining, Inference and Prediction, 2nd Edition, Springer, 2001.
• MacKay, Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003.
• Fukunaga, Introduction to Statistical Pattern Recognition, 2nd Edition, Academic Press, 1990.
• Witten and Frank, Data Mining: Practical Machine Learning Tools and Techniques, 3rd Edition, Morgan Kaufmann, 2011.
• Alpaydin, Introduction to Machine Learning, MIT Press, 2004.

I know there are some good python (pybrain) and ruby (ruby AI libs) libraries out there. The software packages mentioned across the courses include: 

• WEKA and derivatives (YALE, which became RapidMiner)
• R Machine Learning
• Matlab (and Octave the opensource alternative)

The hard part with this project will be a useful algorithm (information processing) based taxonomy, a slight departure from the stats and problem based breakdowns I'm seeing in the seminal texts. I may take inspiration from software libraries more than the way traditional text books are organized. I also want to focus on the algorithm (pseudocode and minimal implementation) like the first book, relegating the theory to the backseat, an approach that is bound to annoy many a grad student.

Finally, I've been looking over Wikipedia (machine learning, list of algorithms), which appears to be a generally useless resource (poor organization and patchy coverage), much like it was with nature inspired algorithms for the first book.