Scientific Product Development: When builders become stakeholders

When the builders of software, the engineers and developers, become the stakeholder in a product, I suspect they initially do a poor job. They are likely ill equipped and treat project execution as yet another exercise optimized for product delivery rather than business sustainability. At least that is what I have experienced.

It has taken me a while to make the connection, but I believe that the corrective measure for those who find themselves in this situation is to rephrase customer expectations as experimental hypotheses to be tested. The trend of for lack of a more suitable name is scientific product development. This is not the software development process alone, this is a broader product management process. It is an empirical approach of using data to validate or invalidate assumptions about what customers want, use, need, will behave, and ultimately what they will pay for.

I think this strategy fits in as a framework for continuous improvement toward business objectives, working in concert with the chosen software development methodology. Although this appears to be the practice for the big guys like google, amazon, and microsoft, there is a push for its adoption in startup's. This is interesting because either it is a case of experienced CTO's commenting on where their experience has lead them (in practice) or where they would like to be (or retrospectively think they were) based on observing the larger players (in theory). I'm not sure which is truth yet.

There is somewhat of an echo chamber of proponents of this movement, particularly as a practice to be adopted by startups. My preferred source at the moment is Eric Ries, who has a blog Startup Lessons Learned which has been live for about four months. He was/is? the CTO of the graphical IM client IMVU and an advisor at venture capital firm KPCB. I've cherry-picked some posts that show the evolution Eric's communication on the topic:

• Ideas. Code. Data. Implement. Measure. Learn: Execute, and as the name suggests focus on the whole process. Drawing on elements from OODA Loop and theory of constraints.
  
• Customer Development Engineering: A presentation on combining agile software development methodologies with concerns of customer development to address the problem and solution uncertainty in a startup.
• The lean startup: An observed trend with startups. Frugal, less waste. Ideas drawn from extreme programming, lean manufacturing, and customer development. A lean startup uses open source, agile methodologies, and learns from their customers.
  
• How to listen to customers, and not just the loud people: Motivated by Seth Godin's Listening to the loud people. Techniques include: using a tracking survey, member's only forum, and customer advisory board.
• The one line split-test, or how to A/B all the time: Controlled experiments like split tests are core to the lean startup. For example: start with a report such as a funnel report and treat it like a cohort study. Use it to challenge your ideas of what customers want, not micro-optimize the product. Focus on macro statistics that are important to the business.
• Thoughts on scientific product development: In response to Mike Speiser's Scientific Product Development. Rather than scrapping features when results don't match expectations, investigate as to why. If it doesn't measurably change customer behaviour, it's not a feature (seems extreme). Less is more for the purposes of limited test variables during controlled experiments.
• What does a startup CTO actually do? A technologist-come-manager perspective on 'technology strategy should serve the business strategy'.
• When NOT to listen to your users; when NOT to rely on split-tests: Listen and attend to users so far as it satisfies business goals. Do not use split tests to micro-optimize, an example of which may be the tuning of persuasive techniques on conversion rates.
• The engineering manager's lament: A great post about the fallacy of the time-money-quality trade-off, the way in while agile methodologies can give all three, and the importance of being rigorous with the adopted methodology (testing to maintain a level quality for example).
• Lean startups vs lean companies: Motivated by Venture Hacks post Lean startups find their moment. Waste in a company (anything that doesn't provide customer value) is different from waste in a startup (anything that doesn't teach you about the customers), particularly as it facilitates innovation.
  
• Principles of Lean Startups, presentation for Maples Investments: A clearer vision for lean startups, focusing on: commodity technologies (open source and platforms), agile software development (extreme programming and test driven development), and customer development (challenging and refining product-market fit). Great IMVU examples in slide presentation.
  
• Using AdWords to assess demand for your new online service, step-by-step: Testing a market using adwords before a product even exists.
• What is customer development?: A summary of the core concept in Steven Blank's book The Four Steps to the Epiphany. Everything you think about your customers is opinion that has to be tested. A process of Customer Discovery, Validation, Creation, and Company Creation.
  

I like Eric's writing style, his perspective rooted in technology, and his message. Much of it seems informed from experience and seems reasonable. Further, his 'three principles' resonates with my recent analysis of my past execution, specifically the need for 'customer development' in addition to an agile methodology and commodity technologies to be a lean startup.

Mike Speiser has a blog Laserlike and he himself has an impressive history of business roles in companies and startup's big and small. Mike has communicated a few posts along these lines recently, the highlights of which were:

• Scientific Product Development: The utility of the scientific method for being systematic in the face of uncertainty, and its use in testing assumptions about customer-feature fit during product development. I the astrology (witchcraft) versus astronomy (science) analogy is amusing and apt.
  
• Using data to make predictions: A general piece about the importance of collecting ad using real data, including finding correlations, testing causation, and the problems with intuition.
  

Some other sources include Jesse's blog 20bits with posts such as Data-Driven Development and Scientific Product Development (as well as other more fundamental topics) motivated by some of the above posts. Venture hacks also interpreted and echoed some of the above posts. I'm sure there are many other sources, but these are the streams I've chosen to been subjected to in my feed reader recently.

As a software engineer and research scientist, adopting the scientific method to address aspects of business uncertainty (an unknown problem and solution) is alluring, perhaps even natural. Although, I can see the potential for misuse, even obsession. It seems clear to me that like an agile software construction methodology, using an customer-focused experimental framework (the so called data culture) for challenging expected customer behaviour is just another tool (however powerful or recently underutilized). One must still have a vision for the product and an understanding of the market for which the product is being optimized.