Friday, August 20, 2010

Chess Rating Machine Learning Competition

For the past two weeks or so I've been hacking on a chess rating machine learning competition at Kaggle. The objective for the competition is to come up with a rating+prediction scheme that can outperform the standard system used for this type of thing called the ELO rating system.

I've been having a good time, hacking together a codebase with a bunch of highly optimizing rating systems as well as a bunch of machine learning techniques. I even managed to reach the top of the leaderboard eariler this week and hold it for a bunch of days. The competition finishes on Monday 15 November 2010, so there is a long way to go yet.

There are a bunch of chess rating systems, and I've had a crack with ELO and Glicko, although there are others like TrueSkill to have a play with. Big wins for me have come for denormalizing the data+highly tuned rating system data and running it through elaborate machine learning techniques.

I've got all my code and data analysis on github and intend to release it after the competition ends (or I get board of it). I've already released a cut down version of Glicko as a demonstration for budding chess rating system hackers.

Some lessons I've re-learned so far in this competition:
  • Unit testing is critical for trust in the code (maths!).
  • A robust test harnesses is critical for offline evaluation (crossvalidation of ideas).
  • An audit trail of experiments and changes is critical (experiment log and source control) 
  • Keep code modular and extensible.
  • Read read read: papers, wikipedia, own notes, graphs of the dataset.
Good times. Kaggle is a great idea (and a Melbourne company!)

Sunday, August 1, 2010

Tricorder or Paperweight?

Get an iPhone 4? The question made me think deeply about my current phone. It's doing it's thing fine and the "shiny new" aspect of the new model hasn't grabbed me. What provoked me was my apathy. I looked at my two year old iPhone 3G not as a phone but as a pocket computer faster than most computers I used or bought for myself until I was at University. It's a tricorder and I'm using it as a paperweight, or at best a walkman.

I run an occasional app, but I'm almost always left underwhelmed. When I first bought it, in particular, when I jailbroke it I was very excited at its potential. A tiny computer that I (and programmers way better than me) could program, that could hit the web - even without wifi. All we had to do was think it up. It was this excitement that tipped me into buying one in the first place.

I occasionally get a resurgence of excitement for all of that potential, but I just don't know how to use it, let along communicate it to those standing around me. It didn't deliver, or the app community didn't deliver, or I didn't deliver. Maybe the expectations were too high (I bought into the marketing) or the device became passe.

This is not a new situation for me. I strongly remember the potential of installing a mod chip in my first generation Xbox - a computer that was in the living room (directly accessible by the humans), that could talk to the network and the Internet, and I (others) could write programs for it - anything we could think up (in theory). I also strongly remember hacking on programming competitions thinking how awesome my solution could be or the amazing experiments I could run with access to a powerful computer, let alone a cluster of compute boxes. Sure that last case may be arrogance, but with EC2 (and similar services) now available and as cheap as ever, I've run exactly zero large compute jobs for this purpose.

I'm less inclined to think that this observation is some form of consumerism side effect and more likely to think that it's a side effect of the accelerating change in technology.

My thesis is:

That creativity prompted by the promise of the new or improved technology evaporates once we are distracted by its reality, once delivered. It becomes passe and we forget our excitement and our ability to dream up creative ways to exploit it (perhaps we even sit comfortably with the way we were told or expected to use it).
I thought I would look back over some key times in my life and project some questions around some of the propose and constraints (I think) I remember having at the time. I prepared a spreadsheet of years, ages, dwelling, and interactions with technology as a prop to help think through these case studies - they're as accurate as I care to make them in the 1-2 hours I've invested for this post.

This section lists some case studies in reverse chronological order (skip to the end if this bit is too self indulgent). Each point lists my age, the approximate year, and I address my self at that time for context and pose some rhetorical questions:
  • Me at 27 y/o (circa 2008): Dude, a website is not a business. A bunch of dudes hacking hard on an idea is not a company. You have passion and drive in spades, but you are misdirected. I know you think you can build anything you can think up, and maybe you can, but you must build something someone wants, where "someone" includes humans beyond you and your hacker friends. Talk to real people in the real world.
    Given 3-6 months and a mixed group of 4 smart programmers and designers who have all committed to your plan, what could you achieve? The barrier to building a piece of software, a service, or a website is very low. Large swaths of (prototype/initial) programming and design is cookie cutter work: mundane and easy to automate or outsource. What are the hard parts (customer development, product/market fit)?
    Consider this: Rather than dreaming about the computation you could achieved with a floor or building full of computers, think about what you could achieve with a floor or building full of employees! How could you direct those minds and harness those resources?
  • Me at 24-25 y/o (circa 2005): Yeah man, a PhD is hard and not for the reasons you thought. Sure, you're off the reservation now, but you're curiosity is a strength not a weakness. You read more papers and dissertations, more broadly than the average cares to read and a lot of it sticks with you, trust me. Also, you may think implementing so many techniques, and automating experiments and statistical tests are not contributions to your research, but they are all invaluable skills you will keep with you and draw upon. Relax dude, you'll get your shit together soon and your productivity will be off the charts, take the time to go slow and learn strange and interesting stuff, because you will never get this freedom again!
    How could you make use of 3 years of paid expenses and no commitments to research your passion? It is an easy idea to acknowledge when starting a research degree, but a very difficult principle to live by during that time. This is that one time to aim for the fences, and failure is OK (as long as its publishable). What would aim to deliver and what impact to the field would you aim to generate?
  • Me at 22-23 y/o (circa 2003): Dude, you can hack systems together, even some pretty complex stuff, but you've lot a lot to learn about engineering maintainable systems compared to what you call programming. Also, being a technological purist is idealistic and naive, there are more constraints than you are considering. You've got the AI bug, that's cool. You've won some competitions, got some accolades - good on you. You may think you want to study AI and be an academic, but only one of those things is true.
    I know those experiments you run, how they execute or days sometimes and you dream about having access to some thumping big computers. If you had access to more CPU and disk than you could utilise at a couple of dollars per hour, how could you make use of it? Beyond experiments and AI programming competitions, what systems would you run on this hardware? What could you tackle (solve? Ha!) with your algorithms and 5, 10, 50, 100 machines?
  • Me at 19 y/o (circa 2000): Dude, you can throw systems together fast which is good life skill but the specifics of CPU/Motherboard/Graphics tech your obsessing about now will quickly evaporate. Network admin - na dude. You're good enough at throwing ad hoc LAN parties together and debugging hardware/software issues, but network admin is not your thing. Keep up the Linux desktop though - it pays off in spades. You know that stupid Java thing you hate, in a few months that shit will snap into place. You'll be hacking together network multiplayer pong games in no time. Seriously dude, programming is your thing!
    So your hacking around with web pages and various scripting languages, that's cool. The ideas you've been force fed about e-commerce are wrong. Consider if the internet became like electricity or TV: a necessity and ubiquitous for the  average joe. Think about what people would need in this environment. Pretend your an engineering/programming savant and the internet pipes are fat and unlimited, what online services do average people in a brave new connected world really need? Sure that new google search kicks ass, but think more broadly. Think more interactive and less-geeky versions of the technical, programming and game (the social) spaces where you currently hang out.
  • Me at 18 y/o (circa 1999): That language your playing around in is ANSI C, you really should consider actually learning its syntax and semantics rather than hacking by trial and error. The lessons you will take away from this time will be the fuzzy stuff (motivating people, finishing what you start, clear communication on the internet, marketing) rather than the technical stuff (your stuff really isn't that awesome). Yeah, you get some press for some of your mod work, don't let it go to your head, and consider the presented opportunities carefully!
    What if you had the programming skills to do more than tweak the games your mod'ing? What could you do if you were able to really change them into anything you could think up? What type of creatures, bots, or even new game styles would you explore?
  • Me at 16-17 y/o (circa 1997): You're getting fast at translating ideas into game levels. Those Quake levels you did were lame, some of those quake 2 levels were interesting, and from memory you had some good vision for those Unreal worlds - but those game engines have their limits dude. Granted your efficient with the tools and you have some creativity, but dude you're no designer. One word: "aesthetics" - you don't have an eye for it. Note, it will be those quick rush job deathmatch arenas hacked together during LAN parties that will out live your passion for level design. This will be an important lesson in (play)testing and iterative refinement with real people. Also, first person shooter game skills have a very short half-life (ha!). Keep an eye on that internet thing, it's a bit cooler than those BBSs.
    3D acceleration is cool right? Well, what if your desktop CPU was more than 12x faster, what could you do if your 3D graphics card was considered it's own CPU (called a GPU) and had more power than a small cluster of dedicated rendering machines? What could you build? What would you harness that power to do? Damn straight, photo realistic rendering, truly massive worlds, abandonment of the the 2D desktop metaphor in favour of a 3D environment... no wait, let's not get ahead of ourselves.
    Also, I know your dreaming about 17-inch or god forbid 21-inch monitors. How would you manage your 2D desktop with 27 or 30 inches of screen real-estate? No, not a square screen - how on earth would that work?
  • Me at 13-14 y/o (circa 1994): Sure, games a fun. Sweet work on hacking the DOS start-up configuration with game-specific memory profiles (autoexec.bat config.sys etc) and an interactive menu system. You may see it as being a necessity, but that grasp of system resources and its wilful manipulation is good kernel of knowledge.
    You know RAM drives you're playing around with - yeah I know you can boot up doom from RAM in seconds rather than minutes from the hard disk. What if I said 4 or even 8 MB of RAM is nothing. What could you do with 4 or 8 GB of RAM? Run all apps from RAM? Run the OS from RAM? What do we need our hard disks for again?
  • Me at 9-10 (circa 1990): Batch files make loading your games faster - good on you. BASIC is not that hard kid, keep at it. Who doesn't like Sim Earth and Arkanoid, but incredible machine and lemmings are cooler.
Alright, so I got a bit carried away with nostalgia there. Good times.

There was the simplest in progression in the limitations and desires: from system resources (RAM, CPU, GPU) to programming power (desktop, internet, speciality compute) to management (time, vision, scope). I don't know if that is a symptom of a single persons journey or the nature of advancement of the time period. If the latter, it suggests that given that we (people of the computer) now have sufficient hardware, market adoption, and software capability, we are left at the pointy end - the open question of figuring out what to do with it.

A more takeaway strategy would be:
To explore the creative impulse and see if there are any opportunities to be exploited when a new or improved technology or service is encountered or a solution for a limitation is desired. To be more methodical, to systematically document, and to potentially participate in delivering on the promise it represents.
This trip down memory lane makes me want to seriously address some of the rhetoric in this post. EC2 is begging for me to melt some CPUs. Heroku and AppEngine are begging for risky web site/service experiments. I look at the now 2 year old tricorder sitting unused on my desk and know that it could be doing more for me.