Reconciling the shift in Software Construction Processes

I started building software to solve my own needs. I wrote scripts to automate monotonous tasks and eventually started modifying source code of games to realise new game modes. There were software engineering and project management classes at University, although the focus of my under graduate degree was implementation.

We were equipped with the foundation of the software development life cycle, requiring intimidate knowledge of the stale process of preparing formal requirement specifications, software specifications, design documents, and even unit test and user acceptance tests. The document-centric software engineering education also included the classical methodologies, including waterfall and spiral as well as their iterative siblings like rational and RAD. We were also exposed to more exotic approaches like CMM, SPICE, and so on.

Software development courses required less formal specifications, focusing on use cases and a suite of UML-centric documentation such as class diagrams, sequence diagrams, and many others. Completing my undergraduate, I still thought solving problems with software was all about writing code, and that pictorial and documentation-centric formalisms were academic or an artifact of more formal construction projects, such as government.

Starting my first job in an analyst role, the facets of computer science and software engineering that were relevant 'in the real world' were not immediately clear to me. As 'the kid', I was relegated in-house projects and prototype work that I executed much like my own side project development work. I'd outline a rough plan focused on the software features, estimate effort and get to work. Slowly, elements of classical software engineering were requested. I'd have to prepare a requirements specification or a unit test plan, sometimes document system operations protocols for more technical work.

I was gifted a copy of Code Complete very soon after starting, and Steve McConnell's best practices in his book were defiantly company dogma. Specifically, documents as a starting point, design documents to motivate well thought out development (ultimately throw away), code reviews, and most of the code specific stuff. After about two years of part-time development, this was my perspective of software engineering, document light, code focused, although in it's own way still a systematic development process, with the important shift in focus (for me) to build for maintainability.

The next two years were very different to the first two, as I was only involved in on-site team projects. I distantly remember the starting of a new senior developer hailing from the UK who started evangelising extreme programming (XP) within the company (maybe the end of 2001). Elements of XP started to seep into our day to day. I remember pair programming on rare occasions during slow times as a strategy to ensure progress on an issue. I think that around this time we adopted (at least I was using) automated builds and automated unit tests, specifically Ant and JUnit because we were mostly an enterprise Java shop. At the time I was excited at the trend of automating tasks with programs over documented procedures, for example mostly dropping deployment documentation and unit test docs. This trend continued to code generation, integrated application servers, and elaborate test frameworks.

I remember a specific project that started in early 2003 that influenced my methodology for constructing software for a number of years after. It was conventional client work (linear) with requirements specifications, design specifications, system test specifications, user acceptance test specifications. We used Rational Rose for design as it could spit out Java code, supposedly jump starting the build. The build was intended to be test-first development (a second within the company I think), where we would write our automated unit tests before the functional code - which occurred for the most part. Looking back, the project was ultimately a classical linear method (waterfall) with sprinklings of XP on top, in an effort to sample those facts that both fit within the traditional expectations of a corporate client and deliver value on the specific project (more careful risk management than risk aversion for the then popular and emerging techniques).

At this point I bailed back to university to complete my PhD in artificial intelligence with this perspective of constructing software, which although I have been tracking broad developments in the space, I had not been active for about three and a half years. I wrote a lot of software as a part of my research. Most of it was scientific in nature, providing prototypes of my ideas and manifestations of other peoples ideas. I call them manifestations because enough experience at reading computer science publications has taught me that the medium is terrible at communicating software artifacts like system designs and algorithms.

I started out writing software with the intention of diligence, of expecting every algorithm and experiment I wrote to have suite of unit tests. It was crystal clear to me as an experienced software engineer, that if I could not trust my software to work as I intended it, than I couldn't trust the results that software produced, no one could. (my work was mostly simulation-based) I quickly learned that not only did many (if any) researches in my broader field (computational intelligence) actively challenge their trust in their experiments with automated unit tests, most had no concern or education in software engineering to speak of. In fact, half way through my candidature I believed (and still believe) that significant progress could be made in the field with systematically developed and reusable algorithm and experimentation platforms, an observation quietly made by a handful of senior researches in related fields over the last decade.

Somehow, like so many others, I lost my way. I released much of my code research product as open source projects, that although offered some basic infrastructure tests, offered a fraction of the diligence I originally envisaged (perhaps ironic given that my girlfriend is doing her PhD in theoretical software testing strategies). I tried to keep any effects of this on my resultant research at the back of my mind. I think the strong software prototype nature and high turn-over of prototypes in my chosen field of research just wore me down, and the siren of measurable progress eroded my diligence.

After submitting my dissertation for examination I jumped back into full time development. I started with some personal projects and later formed a group, focusing on web development with ruby on rails. Rails was (and still is) fascinating to me, I could clearly see familiar things in the lineage of the platform produced by developers with once similar backgrounds to me. Not necessarily an anticipated by-product of the evolution of XP or the umbrella of agile methodologies under which it is defined, but defiantly an animal with many familiar traits accumulating unchecked over the last three years of refinement.

We did not use a raw XP methodology in our group projects, nor did we strictly use the newly emerged techniques of test or behaviour driven development, but we did employ familiar facets. Rather than documentation light, we were documentation weightless, focusing on capturing vague requirements and realising application features. We were self-organizing to the point of being cowboy-esk, unit tests were post-hoc (if at all), and it took us a while to get our (regression-less) continuous integration process going with a staging environment. Many of the other important lessons stuck though. We strove to work in the same space and time as much as possible, promoted end-to-end feature ownership, focused on delivering value to our users (the core of so-called Agile), and iterated with weekly and later daily sprints.

So far, i have missed out on being involved in a pure XP project, and I observed but have not experienced test-driven development (a technique that could be invaluable in scientific computing), and behaviour driven development (a technique that could have been ideal in recent web work given its popularity in the Rails community). I'd be interested in how often such techniques and broader methodology are employed in practice, in corporate work, and in open source. I'm betting the majority of work still operates against linear models, or more likely against linear approaches grafted with value-add techniques from XP.