Setting Up for Success

Who's this for?

This page discusses several generic software development topics.  It is aimed more towards those who undertake software initiatives more akin to a project and less so a product development.  Read more about the distinction here.

Introduction

Digital Product and Software Development has a reasonably high level of inherent risk and uncertainty.  At the risk of dramatically oversimplifying the subject, both Product and Software development, albeit in very different ways, are full of continual problem solving and experimentation to determine what to build and also how to build it.

This page pulls together several concepts to provide some thinking tools for considering how to set up a major new undertaking, such as a completely new product or major feature build, in terms of how much 'upfront' versus 'deferred' analysis and estimating.

Before you interpret this as a waterfall versus agile debate, let's be clear that it's not. Infact one of the reasons this page exists is to consider and describe the situation without using these often misused and misinterpreted terms.

Do we need to do upfront analysis and estimating?

There are many assumptions made about how beneficial upfront planning & analysis is - or indeed is not - in terms of Project Success.  Does it help to set more accurate expectations or is it simply wasting time when we should be cracking on?

The purpose of this page is to summarise some basic scenarios we often find ourselves in.  Scenarios we don't always know how to explain or summarise to our stakeholders, who in turn often don't understand the intricacies and nuances of software development.

We know we don't want to spend excessive time and effort up front on activities that don't directly contribute to software development.  But how do we determine what to front load and what can be done iteratively?  After all, one approach surely isn't the right approach for all situations? 

Well, one way to determine this is to consider the levels of VUCA (volatility, uncertainty, complexity and ambiguity).  High levels of uncertainty or ambiguity requires early activities that explore the issues and help to create clarity and consensus.

The opposite of uncertainty is not necessarily best described as certainty, but instead as knowledgeable or confident.  Reduce uncertainty and increase confidence.

A high VUCA situation may be a medium overall risk situation with a highly competent team.  Conversely, a low to medium VUCA situation may be a very high risk situation with a group of people who are not highly competent.  High risk strongly suggests that it makes sense to prioritise reducing vuca where possible.  Yes, this may well delay some build activities but it should help to unearth risks sooner, which provides more opportunities for effectively managing and overcoming them.

The third consideration is whereabouts in the cone you would describe your current situation and whether assumptions have been made about your current position by others.  This is important in order to know whether there needs to be a bias towards learning through exploration, because the range of uncertainty is too high (and if so how best to do so), or if to bias towards iteratively and incrementally building out the software system.

As Steve McConnell says,

"Meaningful commitments are not possible in the early, wide part of the Cone. Effective organizations delay their commitments until they have done the work to force the Cone to narrow. "

The Cone of Uncertainty showing the ranges in estimation accuracy.

Whilst most literature discusses the cone of uncertainty phenomenon in terms of effort estimation uncertainty, because risks and issues take effort and time to resolve, it also represents the confidence levels of the known risks & issues.  To put it another way, the cone of uncertainty also reflects:

The Back-Pack

It's sometimes helpful to think about uncertainty and ambiguity as a weight, a burden you carry with you... a bit like having a heavy backpack at the start of a long hike.  The fitter and the more experienced you are, the more you can carry.  However, even for these people, losing a bit of the weight is a small change that they will benefit from, throughout the entire hike.

Note that this chart is illustrating the levels of risk and issues being managed and tackled throughout the build.  It is not an indicator of progress.  We all know that the more risks and issues we carry, the more effort is required to react and resolve them, resulting in slower and less predictable progress.

A more representative visualisation of this scenario is to add a range of uncertainty to the level of Risks & Issues as shown here.  Note how the range reduces over time as more work is done and there are fewer hiding places for surprises to lurk.  Note also how the less experienced team fail to reduce the levels of uncertainty as much as the more experienced team.


Read more about the Cone of Uncertainty.

In this situation, even with a highly experienced team, it's often beneficial to invest in some up front work to determine which of these risks and issues are likely to be within the core proposition or if they are in a negotiable scope of value, cost or time.  This type of upfront investment may take the form of activities known as Discovery, Inception or Sprint 0 to name just a few. 

Upfront or Iteratively?

What you decide to do up front will depend on your situation, the degree of uncertainty you're comfortable having, the experience and capabilities of the people involved and many many other factors.  You're effectively considering where the midpoint line is and the range of uncertainty, and determining the best way to get them both to acceptable levels in a timely and cost effective manner.

In the early days of the agile software development movement, before the Agile Manifesto even existed, there was a drive to get rid of the long, upfront planning and estimating activities because they had been shown to be hugely ineffective at forecasting the actual costs, efforts and timescales required.

Instead, Jim Highsmith in particular would discuss the benefits of using some of the project budget - probably about the same amount of costs to run the protracted planning and estimating phases from that era - to start building the software in order to get a better idea of how long it may actually take with this tech, these people and with their tooling.

You would still seek to achieve the same outcome as the traditional way of planning and estimating, i.e. to decide whether a project or product is feasible and viable.  As such, some early developments may be scrapped.  Whilst this can feel wasteful, it is in fact, on average less wasteful because those that aren't scrapped have a head start, the teams and the stakeholders have a better understanding of costs and timescales, and as such lead to a greatly increased project success rate.

And Remember...

Setting up for success is just the beginning.  As Steve McConnell says,

"The Cone narrows only as you make decisions that eliminate variability"