Computational Intelligence in the wind

There seems to be a lot of chatter about genetic algorithms in recent days, for example:

• Genetic A/B Testing with JavaScript: a discussion by John Resig about reverse engineering the Genetify service that seemingly uses a genetic algorithm to drive split testing. Seems like a problematic approach to me, good for sampling the space, but poor for confidence-based decision making over what design to choose.
• CSSEvolve: guided stylesheet evolution: an introduction by Andrew Cantino to CSS Evolve service by IterationLabs that provides an interactive GA for basic sampling CSS designs for a given page (varies fonts, colors, etc). Interesting notion, and likely a good application for an interactive evolutionary algorithm.
  
• Evolution Produces Better Antenna; Casey Luskin Very Upset: on Good Math, Bad Math by Mark Chu-Carroll discussing how some intelligent design evangelist talking about the post of another evangelist, both of which were unhappy by the (awesome) GA-designed antenna work at NASA from 2004 (old news).

For some reason (I was procrastinating), all of this GA talk motivated me to update the wikipedia page for the general and specific fields of research for my doctoral work. I definition and techniques sections as well as cleaned up the Artificial Immune Systems page, and kicked off a new Clonal Selection Algorithms wiki entry. All very basic, but I expect to add more content over the coming days.

Although I've created wiki pages before, this recent bout got me thinking how useful this service could be if it were elaborated by experts in the field. Although it's an encyclopedia, it would be within scope to motivate fields of research, even highlight open problems and provide sample code and test data. Wiki pages could be a useful valuable scientific resources, rather than a #1 Google result that is ignored by those in the field. I'm now thinking how Knol fits into these perspective...

More Video Lectures

I've watched a few more video lectures recently, mostly focused around Complex Systems and Machine Learning. I haven't figured out the best process to internalise from this medium, although I suspect I will progress to something like Peteris Krumins' approach (for example, his analysis of Introduction to Algorithms). Anyway, some notes:

• Stochastic Search Methods by Bogdan Filipič (2005). Provides an introduction into search methods that use randomness, differentiated from calculus methods (like gradient decent) and enumerative methods (like exhaustive search and dynamic programming). The general field is motivated as approximation methods for knowledge discovery in high-dimensional and multi-modal search spaces (where calculus and enumerative methods break down). Focused introduction to Simulated Annealing and Evolutionary Algorithms including Evolution Strategies, Genetic Algorithm, and Genetic Programming.
  
• Dance Evolution by undergraduate students at the Evolutionary Complexity Lab, University of Central Florida (2007). Short demonstration of 3D dancers comprised of neural networks trained with an interactive genetic algorithm, specifically a variant of NEAT. More information and videos available on the project page.
• NERO 2.0: Neuro Evolving Robotic Operatives. An introduction to the nero game for interactively developing controllers for agent simulations using genetic algorithms and neural networks. I remember a keynote on this at a conference back in 2006 - looks like a lot of fun! Official page and project page.
• A Paradigmatic Complex System: The Immune System by Irun Cohen (2008). The presentation of the immune system as a model complex system, managing the subsuming complex system of the host organism. The immune system as computation, cytokine networks, and the system properties being degenerate, pleiotropic, and redundant. The increase in complexity can result in an increase in fragility (invertebrate versus vertebrate immune systems). There was also a great graph showing the trend of complexity against evolutionary time. I'm quite familiar with Cohen's take on the immune system, I researched it for a while for my dissertation, specifically the cognitive paradigm of immunity.
• Emergence of complexity in biological networks: from selection to tinkering by Ricard V. Solé (2008). Considers the pervasiveness of networks in complex systems, and considers the development of such networks under selection (optimization in some cases) and tinkering (working with what is available without foresight). Examples including protein interaction networks and word co-occurrences, each with model systems. The talk pointed me to an interesting paper entitled Evolution and Tinkering (1977) by François Jacob.
• Introduction to Complexity Science by Seth Bullock. Provides an introduction into complex systems with examples including termite mounds, the brain, developmental biology, and evolution. He provides an interesting case for the now pervasive use of evolution as an explanation and/or terminology out side of the field, a role that used to be taken by physics. A good rant at the end about biologically inspired techniques needing to be justified based on their innate suitability rather than their inspiration, tempered with the observation that we can still use tools that we do not fully understand (I ranted on this myself in my dissertation).
  

I have also been thinking about getting into some technical communication. I started with the notion of a book or an ebook providing an introduction to computational intelligence search techniques (something like Thierry Lecroq's book for String Algorithms and associated Java examples). Thinking that the presentation was a bit dry, my thoughts shifted to a blog or mini-ebook based approach (like Peepcode for ruby on rails development). I also think a screen cast medium would be effective for tutorials and demonstrations (like peepcode and railscasts). I want to find a good mix of interestingness, monetizability in a medium to present a set of discrete related topics.

Complexity Research in Australia

I have been doing a little reading in the complexity sciences of late, and have uncovered a network of local (Australia) organisations, projects, and labs focused on this field of research. I presume I had not detected their presence before given my focus on learning and adaptation processes. This subtle influence lead me through adaptationism vs selectionism, selectionist theories, adaptive system formalisms and ultimately to complex adaptive systems, all of which played a critical role in my doctoral dissertation. The difference is that complexity sciences is broader than adaptation by a complex system, where systems can have the attributes of complexity (impervious to reductionism, non-linear interactions between components, emergent phenomena, etc) irrespective of 'improving their performance in an environment' - such concerns are irrelevant in systems such as climate systems, cosmological systems, and other physical systems.

Anyway, I started with Professor David Green at Monash who has a long histroy of complex systems research, including a recent book on landscape ecology looking at networks in ecosystems. Green's lab has setup VLAB, a virtual laboratory for complexity research intended to market some of the more interactive aspects of the field to a broader audience including a large number of in-browser simulations (Java applets).

The Australian Research Council (ARC) appear to have at least three web presences for their interests in complex systems, specifically the Complex Open Systems Research Network (COSNet) that seems to be a network of researches involved in ARC funded complex research, and the ARC Center for Complex Systems (ACCS) hosted at UQ, and the ARC Centre of Excellence for Mathematics and Statistics of Complex Systems (MASCOS). The COSNet site seems the more elaborate of the three, not limited to some insight into the history of the center, its vision that motivates the field of inquiry, and more interestingly example software. I am unclear of the relationship between the ACCS, MASCOS, and COSNet other than an obvious overlap in the source of funding and in some cases center members.

The Commonwealth Scientific and Industrial Research Organisation (CSIRO) also has an interest in complexity, demarcating complex systems science as an emerging science area, and setting up the CSIRO Centre for Complex Systems Science (CSS) directed by John Finnigan. The CSS motivate the field well with specific divisions in biological, physical, and social-ecological systems as well as the fundamentals. The 'emerging science area' site does have some interesting descriptions, presentations, and software downloads, although you have to dig deep into their mesh of pages.

Interestingly some of these organisations have emerged from theoretical physics an atmospheric research, which makes sense given the cross-disciplinary nature of the field and that what are now defined as complex systems are prevalent in such fields. This community seems larger and more motivated in this country (to do: test this assumption) than the field of computational intelligence that I previously chose to pursue, likely given the broader aims of complexity science that clearly have a strong overlap with (encompass?) my field.

Scientific Product Development: When builders become stakeholders

When the builders of software, the engineers and developers, become the stakeholder in a product, I suspect they initially do a poor job. They are likely ill equipped and treat project execution as yet another exercise optimized for product delivery rather than business sustainability. At least that is what I have experienced.

It has taken me a while to make the connection, but I believe that the corrective measure for those who find themselves in this situation is to rephrase customer expectations as experimental hypotheses to be tested. The trend of for lack of a more suitable name is scientific product development. This is not the software development process alone, this is a broader product management process. It is an empirical approach of using data to validate or invalidate assumptions about what customers want, use, need, will behave, and ultimately what they will pay for.

I think this strategy fits in as a framework for continuous improvement toward business objectives, working in concert with the chosen software development methodology. Although this appears to be the practice for the big guys like google, amazon, and microsoft, there is a push for its adoption in startup's. This is interesting because either it is a case of experienced CTO's commenting on where their experience has lead them (in practice) or where they would like to be (or retrospectively think they were) based on observing the larger players (in theory). I'm not sure which is truth yet.

There is somewhat of an echo chamber of proponents of this movement, particularly as a practice to be adopted by startups. My preferred source at the moment is Eric Ries, who has a blog Startup Lessons Learned which has been live for about four months. He was/is? the CTO of the graphical IM client IMVU and an advisor at venture capital firm KPCB. I've cherry-picked some posts that show the evolution Eric's communication on the topic:

• Ideas. Code. Data. Implement. Measure. Learn: Execute, and as the name suggests focus on the whole process. Drawing on elements from OODA Loop and theory of constraints.
  
• Customer Development Engineering: A presentation on combining agile software development methodologies with concerns of customer development to address the problem and solution uncertainty in a startup.
• The lean startup: An observed trend with startups. Frugal, less waste. Ideas drawn from extreme programming, lean manufacturing, and customer development. A lean startup uses open source, agile methodologies, and learns from their customers.
  
• How to listen to customers, and not just the loud people: Motivated by Seth Godin's Listening to the loud people. Techniques include: using a tracking survey, member's only forum, and customer advisory board.
• The one line split-test, or how to A/B all the time: Controlled experiments like split tests are core to the lean startup. For example: start with a report such as a funnel report and treat it like a cohort study. Use it to challenge your ideas of what customers want, not micro-optimize the product. Focus on macro statistics that are important to the business.
• Thoughts on scientific product development: In response to Mike Speiser's Scientific Product Development. Rather than scrapping features when results don't match expectations, investigate as to why. If it doesn't measurably change customer behaviour, it's not a feature (seems extreme). Less is more for the purposes of limited test variables during controlled experiments.
• What does a startup CTO actually do? A technologist-come-manager perspective on 'technology strategy should serve the business strategy'.
• When NOT to listen to your users; when NOT to rely on split-tests: Listen and attend to users so far as it satisfies business goals. Do not use split tests to micro-optimize, an example of which may be the tuning of persuasive techniques on conversion rates.
• The engineering manager's lament: A great post about the fallacy of the time-money-quality trade-off, the way in while agile methodologies can give all three, and the importance of being rigorous with the adopted methodology (testing to maintain a level quality for example).
• Lean startups vs lean companies: Motivated by Venture Hacks post Lean startups find their moment. Waste in a company (anything that doesn't provide customer value) is different from waste in a startup (anything that doesn't teach you about the customers), particularly as it facilitates innovation.
  
• Principles of Lean Startups, presentation for Maples Investments: A clearer vision for lean startups, focusing on: commodity technologies (open source and platforms), agile software development (extreme programming and test driven development), and customer development (challenging and refining product-market fit). Great IMVU examples in slide presentation.
  
• Using AdWords to assess demand for your new online service, step-by-step: Testing a market using adwords before a product even exists.
• What is customer development?: A summary of the core concept in Steven Blank's book The Four Steps to the Epiphany. Everything you think about your customers is opinion that has to be tested. A process of Customer Discovery, Validation, Creation, and Company Creation.
  

I like Eric's writing style, his perspective rooted in technology, and his message. Much of it seems informed from experience and seems reasonable. Further, his 'three principles' resonates with my recent analysis of my past execution, specifically the need for 'customer development' in addition to an agile methodology and commodity technologies to be a lean startup.

Mike Speiser has a blog Laserlike and he himself has an impressive history of business roles in companies and startup's big and small. Mike has communicated a few posts along these lines recently, the highlights of which were:

• Scientific Product Development: The utility of the scientific method for being systematic in the face of uncertainty, and its use in testing assumptions about customer-feature fit during product development. I the astrology (witchcraft) versus astronomy (science) analogy is amusing and apt.
  
• Using data to make predictions: A general piece about the importance of collecting ad using real data, including finding correlations, testing causation, and the problems with intuition.
  

Some other sources include Jesse's blog 20bits with posts such as Data-Driven Development and Scientific Product Development (as well as other more fundamental topics) motivated by some of the above posts. Venture hacks also interpreted and echoed some of the above posts. I'm sure there are many other sources, but these are the streams I've chosen to been subjected to in my feed reader recently.

As a software engineer and research scientist, adopting the scientific method to address aspects of business uncertainty (an unknown problem and solution) is alluring, perhaps even natural. Although, I can see the potential for misuse, even obsession. It seems clear to me that like an agile software construction methodology, using an customer-focused experimental framework (the so called data culture) for challenging expected customer behaviour is just another tool (however powerful or recently underutilized). One must still have a vision for the product and an understanding of the market for which the product is being optimized.

Persuasion Science on the Web

I recently made the time to listen to the NPR podcast entitled The Science of Getting A 'Yes' that interviews Robert Cialdini about his new book Yes! 50 Scientifically Proven Ways to Be Persuasive (2008).

The subject is very compelling, especially in the context of controlled experiments on the web as a motivator for testing assumptions about customers. I found the anecdotes from related scientific studies very interesting. I captured some notes:

• Consider communicating what might be lost by not adopting the product as opposed to what might be gained (prospect theory). For example, the potential harm of not breast feeding rather than the benefits a breastfed child may receive (from a caller to the show).
• Potential danger of negative social proof. For example, signs to not steal petrified wood from a forest increased theft, announcements not to cheat on taxes increases cheating (perceived that peers are acting in a particular way promoting that behavior). An interesting counter example was that of the reuse of hotel towels, where signs that promoted the number of people that reused towels were far more effective at encouraging that behaviour than generic slogans about helping the environment.
  
• An unexpected and personalised gift. For example, a waiter received more tips by giving one mint per diner with the cheque. More tips again with two mints per diner, and more tips again with one mint and a second round of mints provided 'just for those customers'.
• When asking for something use 'because' an suggest a reasonable reason for whatever is being requested. People generally want to reciprocate, they just need a reason.
  

The book has an official website, that provides podcasts and extracts. I stumbled across a more complete list of the 50 take-away points of the book, which although are bland, are still provocative for web-based experimentation. There is little doubt that this is a rich source for marketing optimizations, although the shear number of points makes it hard to internalize. I think a good strategy is to take a given product and go through the list assessing the product in the context of each point (where appropriate).

Cialdini also has another book entitled Influence Science and Practice (2008, 5th edition) that has a similar theme. The wikipedia page for the book seems to summarise the core message to a series of effects referred to as the 'weapons of influence', which are: reciprocation, commitment and consistency, social proof, liking, authority, and scarcity.

This form of marketing (exploiting sociology research) does feel somewhat insidious. For example the excerpt provided with the NPR interview provides an example of the success of a telemarketing campaign that was attributed to changing "Operators are waiting, please call now" to "If operators are busy, please call again", a clever use of social proof. I guess persuasion by definition is a manipulation of others, and the ethical use (let alone optimization) of these tools appears subtle. It is just word play on the web after all, although I suspect such approaches could be (are?) a gateway to more devious social engineering.

Refining a Project Execution Strategy

Historically, my projects have been designed to satisfy my own needs. Their utility to others after their public release as open source or similar was incidental. Projects predominantly resulted in software or a document as a the work product involving development or research to complete. Research work focused on addressing a given research question, most commonly centered on an expanded understanding of a specific area or technique. Software work was always centered around a need manifest in a loose set of requirements and design notes, less rigorous than my professional software development endeavors.

I would document and elaborate only so far as the process would refine my understanding of the proposed product sufficiently that I could push out a prototype as quickly as possible. I remember reading Atwood's post in late 2007 on software vision statements that motivated me at the time to sharpen the need for each of my side projects. Specifically I adopted the formulaic vision statement taken from Geoffrey Moore's book Crossing the Chasm, which worked well for the open source projects I was working on at the time.

When our group formed earlier this year, we quickly adopted the forum-structure of basecamp to manage our internal ideation. It seemed to work well for us, and we soon had a long list of project 'posts', each with an active comment stream refining the idea and understanding of the market. I think our first problem was the motivation behind our project selection criteria. The selection of our first project was based on a lose set of factors which boiled down to our collective intuition of which project had the best chance of success. Unfortunately even this process was abandoned on subsequent projects, which were decided pretty much on whim.

We were a team of builders, all developers or designers used to the daily grind of actually building software, and naturally this is exactly how we treated our own projects. We were students of the web authorities on technology startups, and I think we accumulated advice through our builder perspective. We initially knew we needed a monetisation plan (at least for the first project), although our second core weakness was that we were of the collective opinion that implementation was the most important aspect ("build it and they will come").

We identified what we discerned was a market and an opportunity, but we stopped short of assessing who comprised the market and what their need was, let alone refining that understanding over the iterative development of the product. We were experienced builders and we had little difficulty executing the implementation of whatever we dreamed up, although the pattern of primitive initial market research and a focus on the product was repeated for each project we executed.

I think these are absolutely crucial observations for engendering future improvement, especially given that I expect I was a large contributor to our past execution pattern.

The lack of a project vision was apparent to me when I came to write bio's and descriptions of each project for our team page. The warning signs were there, each of us always did a poor job of explaining our projects to anyone that shows a slightest bit of interest. We always knew what we were building at the time, although we were never able to reel off a clear pitch. We didn't make the time to clearly define who the product was for and what value it provided to them.

The focus on team and product over market has taken much longer for me to detect. I was looking at our projects that we had declared successful (because we had delivered), and thinking that our measure of success was faulty because user uptake and purchases remained were not growing. The clinching moment occurred recently while re-reading Marc Andreessen's guide to startups series from last year, specifically that the only thing that matters is market. It wasn't that we were a great team or that we had great products, it was we knew very little about each product's market beyond a superficial survey, and as such did not consider (let alone measure or attempt to refine) product-market fit.

A final important realisation is that as a team of builders we are used to having clients and stakeholders to go back to for ground truth on what we are building. Someone with the money enlisting us to realise their vision or need. Scratching our own (or a perceived) itch meant that we were our own stakeholder as well as development team, making the waters very muddy when arguing feature sets and user requirements. Most critically, as developers we saw product delivery and launch as the hallmark of a successful project, although our broader company goal was to create a self-sustaining business, clearly a mismatch.

Business, startup's, and 'the process a group of hackers use to building websites with the hope of turning them into businesses' is not a science. I read over and over again how nobody really has a clue, how much of the advice is personalized and retrospective making it hard to abstract patterns. 'Give it a shot' and 'build something people want' are great slogans for capable builders, I think as long as there is a savvy mentor helping to guide the ship or there are periods of self-reflection followed by self-education by the team.

With thoughts of self improvement in mind, the following are some techniques I'd like to experiment with in future project execution, perhaps an augmentation of our until now product-focused execution strategy:

• Internal Product Plan: A real and formal document that contains at a minimum the vision for the product, an elaborated description of the product features, monetisation plan, and preliminary market research (market assessment, existing product feature comparisons). This is the 'what are we doing and why' (likely a lightweight business plan), the basis for elevator pitching to anyone that will listen and more importantly a process that forces (some) deeper thought into the need for the product early on.
  
• Product Manager: Someone to take ownership of the holistic product. An internal client to be responsible for learning the market and to measure and guide the adaptive fit of the product. The project manager role would likely be a developer splitting their time, meaning the market research tasks would have to be intelligently selected based on their expected benefit.
  
• Measuring and Refining: There must be processes of measuring and improving the understanding of the market, the need for the product, and the product-market fit. For example: pre-build market tests, deeper understanding of chosen monetisation plan, and continuous conversion optimization and data-driven feature management using controlled experiments.
• Priority Queue-Driven Development: As builders that have been around the block execution is an ease, especially with foreign technologies and ill-defined requirements. It's just part of the job. Although I've ranted about software construction processes, I would keep our build process much the same as it is, although with the addition of a disciplined priority queue-based feature implementation strategy (based somewhat on Yegge's 2006 post on the practices in Google). We use trac although only as a bug tracker and feature request ticketing system. I think we could use it or a similar system to dynamically manage a suite of user stories, use comments to debate their merit, and severity to rank their ordered self-organizing consumption and completion by developers.
  

Rather than a complete execution strategy or required tasks, I'd like to experiment with facet of these techniques to assess their benefit. I think a potential problem in this particular case of execution analysis is the danger of swinging concerns too far away from product and team into business and market, simply because they have been relatively under addressed thus far. I presume level heads, metered adoption, and continuous internal questioning would help, if we can muster the diligence.

There are further concerns that remain out of scope at this time like hiring and more detailed customer development simply because we have not had the volume and thus the need to address them, although I look forward to the day.

Documentary on Complex Networks

I finally made time to watch the ABC documentary on complex networks entitled "How Kevin Bacon Cured Cancer" after being promoted on the science show and edge.

The focus of this popular science piece is on small world networks and a brief on the scientists that launched the field of network theory. The evolution of the field is told to the story of an experiment to recreate the classical six degrees of separation experiment.

The documentary wasn't a bad way to spend a lunch break, although their experiment appeared to produce poor results where only 3 of the 40 initial letters made it to their destination ("three chains" with an average of six steps). It also felt like the editors had explicitly trimmed scientific terminology, even the word scientist. Whatever. I noted the core players in the narrative, as follows:

• Steven Strogatz: initially interested in the system dynamics of human relationships
• Duncan Watts: (Australian), student of Steven, addressing the issue of synchronicity (population of unique individuals cooperating). With Steven studied and described small world networks (very clustered, few random links), examples included Hollywood actor graph, US power grid, neuron connectivity in the brain, seminal paper
• Albert-László Barabási: added a important missing piece regarding the structure of the networks - the notion of connection hubs, concerned with scale free networks, devised classical equation to describe these networks (power law relationship of edges, for example), examined the web
• Alessandro Vespignani: interested in diffusion, focused on the use of such networks to transport things, spread of disease (like AIDS), target hubs to combat the spread of disease (for example), application of such work was claimed to be useful for locating terrorists on social networks
• Marc Vidal: the recipient in the small world experiment in the documentary, interested in mapping the network of protein interactions with the cell, mapping networks of the genes involved in disease
  

I remember reading into the field of complex networks (small world and scale free in particular) in 2005 because I had a colleague who was investigating such structured in evolutionary algorithms. I ended up hacking together variations on the self organizing map as an adaptive coverage model using small world structures - the work didn't go anywhere.

The best paper I cam remember from that time for jump-starting an understanding of the field (handed to me by my colleague) was Mark Newman's 2003 review article "The structure and function of complex networks".

Controlled Experiments on the Web

I have been thinking a lot about statistical hypothesis testing on the web recently, likely given the popularisation of A/B testing (more specifically, people talking about the approach). Anyway, I came across a great talk by Ron Kohavi from Microsoft Research (formally Amazon) entitled "Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO" (2007).

My notes from the watching the video:

• HiPPO: the guy with money who makes decisions (Highest Paid Person's Opinion)
• Case - Amazon shopping card: test whether showing recommended items when a product is added to a basket promotes larger basket size and increased cross-selling or decreases conversion due to distraction. Not expected to work, wildly successful.
  
• Case - Checkout page: test two version of a check out page to see which results in more conversions. New version was no good, voucher code distracted customers from completing purchase
• Case - Office online help articles: changing 'was this helpful' message to stars and reason. Simple version was worse (response rate), stars with dynamic text box much better, dynamic box with customized messages much better again.
• Case - Selling Sewing machines: test crazy ideas of selling two machines with 10% off. Tried anyway and resulted in an increases in sales.
• Data trumps Intuition: the less data the stronger the opinions (guesses), collect data through experimentation. Intuition is poor.
• Define OEC: what to measure (Overall Evaluation Criteria), hard to define, not click through but sales for example. Selection of criteria to Optimize for customer lifetime value. Measure lots of things for post-hoc data mining.
• OEC defines whether or not a feature is launched. Drill down into the data with neutral or negative results, attempt to quantify the why, isolate segments, even find bugs in implementation.
• Controlled Experiments: classical statistical hypothesis testing, control (existing version) and treatment (new version), randomized experimental design (strategy for eliminating variables)
• Advantages: find causal relationships (correlation is not cause), insulate external factors (randomness)
• Problems: hard to agree on what to measure, what is being optimized (OEC), quantitative results do not explain why, Primacy effect (experienced user expectations), multiple concurrent experiments (potential of confounding results - interaction), inconsistent trial assignment (cleared cookies if cookie based, etc), not used during critical periods (press)
• Measure Variance: run A/A test (two populations on control) to measure standard variance in the data, run at same time as A/B, avoid conclusions based on random variation (there are statistical tests for this - confidence intervals), power calculations (minimum sample size)
• Strategy: run 50/50 to get peak efficiency, run small for long period, run small and scale up (ensure consistent trial assignment), cancel test if treatment is clearly worse than control (big problems appear even in small samples)
• Trial assignment is important: consistent, independent experiments, consistent ramp up
• Statistics: take the time to execute the more complicated equations, increase reliability, computer speed - computation is what computers are for (lots of books)
  
• Automated systems: make it easy to run experiments, data-driven culture, near real-time optimizations, automate widget placement - automated A/B testing (data driven selection of best content)
• Lessons: listen to customers, intuition is terrible, replace HiPPO's with OEC, careful and accurate statistics, experiment often to optimize value

Grab the slides, the conference paper (official site), an expanded version of the paper. The paper provides some useful details around the specific terminology and more interestingly around real-world implementation and techniques used in scaling of random treatment allocation. The Microsoft-based EXP platform has a few interesting resources such as their excel-based statistical power calculator, and a collection of their talks and papers.

The most thought provoking aspect for me was the automation of A/B testing to the point of real-time selection of content for Amazon's home page. Feels like a statistical version of the human-voting systems like Digg, Reddit, and Hacker News. I'd love to through some generative algorithms into the mix to dynamically vary treatments towards fine-grained optimizations (automated feature hill climbing).

I get the feeling that such methods are standard practice at big places like Amazon, Microsoft, Google, eBay, etc. Although I'm sure there will be a lot of knowledge locked up in these organisations, I bet there are many more papers, talks, even books on the subject. A good place to start are the references in the paper. Some interesting links include:

• Early Amazon: Shopping cart recommendations (2006) the amazon card recommendation story told by the guy involved Greg Linden
• How to Decrease Sales by 90 Percent (2003) an important lesson about making small moves (not changing too many things) between treatments and incrementing
• How to Increase Conversion Rate 1,000 Percent (2003) example of a coupon code causing problems and its systematic identification
  
• Design Choices Can Cripple a Website (2005) highlighting the effects design decisions can have on a web page's call to action
  

Reconciling the shift in Software Construction Processes

I started building software to solve my own needs. I wrote scripts to automate monotonous tasks and eventually started modifying source code of games to realise new game modes. There were software engineering and project management classes at University, although the focus of my under graduate degree was implementation.

We were equipped with the foundation of the software development life cycle, requiring intimidate knowledge of the stale process of preparing formal requirement specifications, software specifications, design documents, and even unit test and user acceptance tests. The document-centric software engineering education also included the classical methodologies, including waterfall and spiral as well as their iterative siblings like rational and RAD. We were also exposed to more exotic approaches like CMM, SPICE, and so on.

Software development courses required less formal specifications, focusing on use cases and a suite of UML-centric documentation such as class diagrams, sequence diagrams, and many others. Completing my undergraduate, I still thought solving problems with software was all about writing code, and that pictorial and documentation-centric formalisms were academic or an artifact of more formal construction projects, such as government.

Starting my first job in an analyst role, the facets of computer science and software engineering that were relevant 'in the real world' were not immediately clear to me. As 'the kid', I was relegated in-house projects and prototype work that I executed much like my own side project development work. I'd outline a rough plan focused on the software features, estimate effort and get to work. Slowly, elements of classical software engineering were requested. I'd have to prepare a requirements specification or a unit test plan, sometimes document system operations protocols for more technical work.

I was gifted a copy of Code Complete very soon after starting, and Steve McConnell's best practices in his book were defiantly company dogma. Specifically, documents as a starting point, design documents to motivate well thought out development (ultimately throw away), code reviews, and most of the code specific stuff. After about two years of part-time development, this was my perspective of software engineering, document light, code focused, although in it's own way still a systematic development process, with the important shift in focus (for me) to build for maintainability.

The next two years were very different to the first two, as I was only involved in on-site team projects. I distantly remember the starting of a new senior developer hailing from the UK who started evangelising extreme programming (XP) within the company (maybe the end of 2001). Elements of XP started to seep into our day to day. I remember pair programming on rare occasions during slow times as a strategy to ensure progress on an issue. I think that around this time we adopted (at least I was using) automated builds and automated unit tests, specifically Ant and JUnit because we were mostly an enterprise Java shop. At the time I was excited at the trend of automating tasks with programs over documented procedures, for example mostly dropping deployment documentation and unit test docs. This trend continued to code generation, integrated application servers, and elaborate test frameworks.

I remember a specific project that started in early 2003 that influenced my methodology for constructing software for a number of years after. It was conventional client work (linear) with requirements specifications, design specifications, system test specifications, user acceptance test specifications. We used Rational Rose for design as it could spit out Java code, supposedly jump starting the build. The build was intended to be test-first development (a second within the company I think), where we would write our automated unit tests before the functional code - which occurred for the most part. Looking back, the project was ultimately a classical linear method (waterfall) with sprinklings of XP on top, in an effort to sample those facts that both fit within the traditional expectations of a corporate client and deliver value on the specific project (more careful risk management than risk aversion for the then popular and emerging techniques).

At this point I bailed back to university to complete my PhD in artificial intelligence with this perspective of constructing software, which although I have been tracking broad developments in the space, I had not been active for about three and a half years. I wrote a lot of software as a part of my research. Most of it was scientific in nature, providing prototypes of my ideas and manifestations of other peoples ideas. I call them manifestations because enough experience at reading computer science publications has taught me that the medium is terrible at communicating software artifacts like system designs and algorithms.

I started out writing software with the intention of diligence, of expecting every algorithm and experiment I wrote to have suite of unit tests. It was crystal clear to me as an experienced software engineer, that if I could not trust my software to work as I intended it, than I couldn't trust the results that software produced, no one could. (my work was mostly simulation-based) I quickly learned that not only did many (if any) researches in my broader field (computational intelligence) actively challenge their trust in their experiments with automated unit tests, most had no concern or education in software engineering to speak of. In fact, half way through my candidature I believed (and still believe) that significant progress could be made in the field with systematically developed and reusable algorithm and experimentation platforms, an observation quietly made by a handful of senior researches in related fields over the last decade.

Somehow, like so many others, I lost my way. I released much of my code research product as open source projects, that although offered some basic infrastructure tests, offered a fraction of the diligence I originally envisaged (perhaps ironic given that my girlfriend is doing her PhD in theoretical software testing strategies). I tried to keep any effects of this on my resultant research at the back of my mind. I think the strong software prototype nature and high turn-over of prototypes in my chosen field of research just wore me down, and the siren of measurable progress eroded my diligence.

After submitting my dissertation for examination I jumped back into full time development. I started with some personal projects and later formed a group, focusing on web development with ruby on rails. Rails was (and still is) fascinating to me, I could clearly see familiar things in the lineage of the platform produced by developers with once similar backgrounds to me. Not necessarily an anticipated by-product of the evolution of XP or the umbrella of agile methodologies under which it is defined, but defiantly an animal with many familiar traits accumulating unchecked over the last three years of refinement.

We did not use a raw XP methodology in our group projects, nor did we strictly use the newly emerged techniques of test or behaviour driven development, but we did employ familiar facets. Rather than documentation light, we were documentation weightless, focusing on capturing vague requirements and realising application features. We were self-organizing to the point of being cowboy-esk, unit tests were post-hoc (if at all), and it took us a while to get our (regression-less) continuous integration process going with a staging environment. Many of the other important lessons stuck though. We strove to work in the same space and time as much as possible, promoted end-to-end feature ownership, focused on delivering value to our users (the core of so-called Agile), and iterated with weekly and later daily sprints.

So far, i have missed out on being involved in a pure XP project, and I observed but have not experienced test-driven development (a technique that could be invaluable in scientific computing), and behaviour driven development (a technique that could have been ideal in recent web work given its popularity in the Rails community). I'd be interested in how often such techniques and broader methodology are employed in practice, in corporate work, and in open source. I'm betting the majority of work still operates against linear models, or more likely against linear approaches grafted with value-add techniques from XP.

Nostalgia for Adaptive Systems

After posting my thesis I felt a little nostalgic for the area of research for which I'd invested nearly four years. I went in search of a fix today, browsing around the impressive collection of videos on VideoLectures.net, and stumbled across a lecture on evolution and complex adaptive systems entitled "Adaptive Behaviour and Emergence" by Seth Bullock.

Seth's introduction of complex and adaptive systems was tight, reminding me of the relationship of this field with my own doctoral research and the months I spent climbing into it. Specifically, while taking notes, I noticed that the packaging of the field and the way it was presented was much the same as they way I discovered and navigated the material on my own, pushing out technical reports on: Complex Adaptive Systems (core CAS theory), Satisficing, Optimization, and Adaptive Systems (confusing influence of optimizing selectionist models), An Adaptive Systems Formalism (core adaptation theory), and Darwinism and Selectionist Theories (units of selection). These threads motivated large aspects of my research and appeared in my thesis in various refined forms.

Anyway, I thought I would capture my notes with links to motivate anyone generally interested in the topic to check out the video lecture.

• Dealing with complexity: system level behaviours (emergent), differentiate complex systems (weather), from complex adaptive systems (brain), from mixed systems (ecosystems). Differentiate nomenclature adaptation, adapt, adapted, etc
• Study of Complex and adaptive systems: spans domains, core features that such system exhibit such as scale, connectivity, nonlinear interactions. adaptive systems are typically complex, but not necessarily the reverse
• Units of selection: genes as replicators (classical Dawkins), genes as information, factors include: longevity, fecundity, fidelity of copy. Persist across generations. Gene focus dissolved the siren of group selection - the adaptive fit for the good of the group (species, sub-species group, ecosystem, etc)
  
• Game Theory: borrowed from economics and international relations, resulted in huge progress in the study of evolution. Classical games from such work that have stuck: tragedy of the commons, prisoners dilemma, hawk-dove. Lesson: short term gains do not pay-off.
  
• Evolution appears progressive: Another trap. Idea of evolution as a force for progress, evolution != optimization (adaptationism), apparent increase in complexity, simple models to demonstrate random walks can demonstrate similar properties, evolution as directed change with positive effect (for who?)
• Arms Race: addressing progressive arguments, more classic Dawkins, notion of local progress, no necessary winners, no long term strategies or goals, short term progress
• Major Transitions: more on evolution as progress, transition events that introduce major changes, open up new opportunities, new niches, can go back (be lost) but typically do not, Dawkins' proposition of watershed events and their preservation as a one-way ratcheting (examples like the cell, multi-cellular, etc)
• Conceptualization: fitness landscapes, from optimization perspective, hill climb landscape of potential arrangements, genetic operators and resultant neighbourhoods are far more complex in reality
• Open-ended evolution: fitness landscapes bound evolution, complexity perspective prefers an open ended perspective, reliance on an environment under which adaptation is promoted, replicators capable of creating more complex replicators (grammar)
• Apparent Design != Evolution: structures that seem as as though they were designed may not be adaptations, they may be the result of natural processes (physics), for example ordered structures, galaxies, solar systems, snow flakes
• Scope: Physics defines the scope of stable forms as well as evolvable forms, constrains evolutionary processes
• Self Organization: Introduces the importance of self-organization to complexity theory and adaptive systems, classic Kauffman self organizing structures as the seeds for natural selection (morphospace)
  
• Edge of chaos: complex adaptive systems cannot be either too stable or too chaotic, sweet spot 'on the edge of chaos', classic Langton
• Power Laws: scale free properties of complex systems, difference between normally distributed and power law distributed, solar flares, earth quakes, classic self organized critically
  

You have got to love Wikipedia these days, it has so many technical topics!

I enjoyed Seth's direct presentation of the material and expect to watch some of his remaining video lectures that seems to focus around simulation and evolution, not surprising given his research into simulated evolution.

Dr Jason Brownlee

I am proud to announce that I have completed my PhD, dropping off the final bound copies today. I'd like to sincerely thank my adviser Professor Tim Hendtlass, and my two examiners Professor Jon Timmis and Dr Emma Hart.

Topic:
Clonal Selection as an Inspiration for Adaptive and Distributed Information Processing

Abstract:
The development of adaptive and intelligent computational methods is an important frontier in the field Artificial Intelligence given the fragility of top-down software solutions to complex problems involving incomplete information. This dissertation describes a systematic investigation of the Clonal Selection Theory of acquired immunity as a motivating information processing metaphor of a series of adaptive and distributed Computational Intelligence algorithms. The broader structure and function of the mammalian immune system is used to frame the cellular theory and classically inspired approaches, providing the additional distributed perspectives of a 'host of tissues' called the Tissue Paradigm and a 'population of hosts' called the Hosts Paradigm. This investigation was motivated by three open problems in the broader field of Artificial Immune Systems, specifically the perceived impasse in the development, identity, and application of the field, the promise of distributed information processing, and the need for a framework to motivate such work.

The state of Clonal Selection Algorithms is investigated in the context of immunological theory, and considered in the context of broader related machine learning fields and adaptive systems theory. A systematic approach is adopted in considering the adaptive qualities of clonal selection beyond a cellular perspective, involving the identification of the lymphatic system and lymphocyte migration as a motivating metaphor for intra-host distributed systems, and host immunisation and evolutionary immunology as a motivating metaphor for intra-population distributed system design. Relevant immunophysiology and theory was reviewed, abstracted to computational models and algorithms, and systematically assessed on model pattern recognition problems to demonstrate and verify expected information processing capabilities. The empirical investigation reveals a variety of tissue and host based clonal selection systems capable of acquiring distributed information via internal processes of controlled localisation and dissemination, in a decentralised information exposure environment.

The general capabilities of Clonal Selection Algorithms as a Computational Intelligence paradigm are defined in the context of a detailed assessment of the suitability of the approaches to the important problem primitives of Function Optimisation and Function Approximation. The findings highlight the general capabilities of the approaches as mutation-based parallel hill climbers for global optimisation and prototype quantisation approaches to function approximation. Finally, an Integrated Hierarchical Clonal Selection Framework demonstrates the subsumed relationship between the cellular, tissue, and host classes of algorithm, the dependent relationship with the complexity of the abstract antigenic environment addressed by the system, and a general scaffold for a broader class of distributed artificial immune systems.

Download:
Thesis (pdf 5MB)