Getting over the Waterfall — Kevlin Henney
pdf-download epub-downloadHowever, it is all too easy to mock the waterfall approach without understanding its motivation and attraction; without such understanding, advocacy and adoption of agile development can be superficial, dogmatic and propaganda based, which misses much of the deeper rationale and actual value of agile approaches. It is not my intention to defend, let alone advocate, sequential development as a general approach, but any honest criticism of it in favour of something else needs to begin with a more even-handed appraisal.
The organising principle of sequential development can be summarised quite simply: strictly align development activities with phases in development. This principle is tidy, clear and easy to explain: given a number of different activities that occur in development, combined with the recognition that a development effort follows a lifecycle that can be characterised differently at each point in time, define each phase of time in terms of an activity and run the phases in sequence. Such a model of development is easy to lay out and present visually. At any given point in time it is clear what activity is being performed. The sequence of activities seems reasonably organised so that problem discovery and reasoning comes before problem solution and execution, which in turn precedes final confirmation that the right thing was built right before proceeding onto deployment.
This model is entirely logical if you make some important assumptions: the problem being addressed is stable and fully understood by all parties (in the same way); the approach to the solution is well defined and fully understood by developers; the technology to be used is fully understood by developers and its use is guaranteed to be free of surprises. If you can guarantee those assumptions, you can ride the waterfall and keep your head above water. But if you can't, the raft of assumptions quickly unravels into something far less watertight.
Software development is typically a multi-variable problem with few guaranteed constants. Treating a dynamic situation with a static plan is a recipe in risk, easily upset by the slightest change. The waterfall approach has the laudable intent of attempting to derisk unknowns by exploring the problem in detail at the start of the lifecycle, with a final check on things at the end. Unfortunately, in practice the effect can be quite different: instead of derisking at the earliest possible opportunity, this approach pushes and accumulates risk towards the back end of the lifecycle. Significant decisions are made at the start, the point of least knowledge of what is involved in developing a system, both in terms of tangible requirements and technical requirements. At the point of greatest knowledge - the end - the chance for effective change has all but vanished. This is not to say that there is no merit in emphasising problem discovery and architectural foundation early in development, just that these activities are not the exclusive preserve of the front end of the lifecycle and their results are not set in stone. It is precisely because of the uncertainty surrounding these issues that you want to start them early. The reason you engage in a development lifecycle with repeating feedback loops is to give yourself the opportunity to clarify and converge as you go, replanning and redesigning as you learn rather than being caught off guard when you're supposed to be done.
It is perhaps telling that, in response to the publicity surrounding agile development, some advocates of waterfall-style development have favoured a rebranding of sequential process models under the heading "plan-driven development". The intended implication being that agile development is unplanned. However, a more accurate reading is that an agile development lifecycle is not driven by a plan, and it turns out this is not wrong: agile approaches tend to be highly planned or, more accurately, "planning driven", but not "plan driven" - a subtle but important distinction. For the reasons examined, being driven a plan is a fragile and risky approach when what are assumed to be constants are actually variables. The role of the term "plan" in agile processes is that of a verb rather than a noun: planning is an activity that is pervasive and continuous, not a static artefact produced at an early stage as input to later phases. It is planning rather than the plan that takes centre stage in agile development. The metaphorical entailment of "plan-driven development" is perhaps closer to the idea of "planned economy" than anything else - and makes the majestic, natural imagery of a waterfall somehow more attractive.
That said, it would be disingenuous to say that plan-driven approaches exclude the possibility of modification or revision to an initial plan in the light of new information, changed circumstances and measures of progress. The attitude to such changes, however, is that they are corrections, irritations and exceptions rather than the normal state of affairs. In spite of much published wisdom to the contrary, estimates are still often treated as predictions rather than as forecasts. Some amount of change and uncertainty is acknowledged, but it is a grudging and partial acceptance that does not inform the overall mindset or nature of the development process. Given the typical human response to change and uncertainty, this is hardly surprising: this trait is within each of us to a greater or lesser extent.
All this suggests that the management of incremental development, particularly processes intended to be streamlined and responsive, is not necessarily the path of least resistance. It sounds hard: it is continuous; it is constantly buffeted by change; it can come into conflict with human nature. The apparent alternative of following a plan that lays everything out in a predictive and tidy sequence, abstracting out interference from change, discovery and human nature, does indeed look simpler. But appearances can be deceptive. Quoting from the last column ("Down on the Upside", ObjectiveView 9): "A good abstraction is one that allows us to develop a piece of software more effectively; a poor abstraction is one that misleads us by omitting or including the wrong kind of detail". In this case, the appeal of the highly planned model comes from abstracting away some fairly critical details - details that if taken into account would, by necessity, change the needs and nature of the development process. So yes, the management of an agile process sounds hard and it is, but that's a property of software development rather than specifically of agile approaches; using a mismatched model of software development makes the challenge of management even harder. Where jumping a waterfall requires a leap of faith, agility is more openly feedback driven and evidence based, using smaller steps to ensure footing and gauge the next step.