The Inconvenient Truths of Software Development

Cartoon by Clay Bennett

Software development has often been managed using a predictive planning model based on the following principles.

The Comforting Illusions

  1. Software development is best managed as a set of projects, with each project starting and ending at specific points in time.

  2. There exists a finite set of requirements for any software development project.

  3. That finite set of requirements is discoverable within some reasonable period of time.

  4. That set of requirements can be accurately documented and validated before any working software is produced or executed.

  5. Requirements are most usefully considered in a pure state, untainted by any traces of design.

  6. The clearest way to communicate requirements is through documentation containing some combination of text and diagrams.

  7. Requirements can be reliably validated through careful review of this documentation.

  8. Customers can be relied upon to accurately identify the relative priorities of their requirements.

  9. It is possible to define a repeatable process for requirements discovery, based on the principles above.

  10. The goal of design is to satisfy all the requirements.

  11. It is very expensive and risky to change software once it is coded and working, so it is critical to get the requirements and design right before coding is allowed to begin.

  12. At some point in the project early enough to be useful for purposes of planning and governance, the amounts of time and effort that will be required to complete the project can be accurately estimated.

  13. As with other engineering disciplines, software engineering is based on a stable foundation of immutable laws.

  14. It is possible to accurately gauge the overall progress of a project by collating and summarizing progress data from project team members.

  15. Projects based on these principles will tend to produce satisfied customers.

These principles sound reasonable. What’s more, they cohere nicely, coming together to form a picture of software development that seems pleasingly rational and predictable.

Unfortunately, the general consensus of actual software developers with real-world experience is that all of the above principles are wrong. In fact, taken together, they can be viewed as a convenient fiction.

In practice, all of the following have been repeatedly demonstrated to be true.

The Inconvenient Truths

  1. Useful software products evolve rapidly and more or less continuously, so defining a “project” as a series of development activities with definite start and end dates is typically somewhat arbitrary, and often misleading.

  2. Software requirements are a matter of opinion, and are therefore subjective, largely unbounded, always incomplete, and open to perpetual redefinition and reinterpretation.

  3. Requirements documents for software development efforts of any significant size are generally difficult to understand and interpret consistently, and so offer an entirely misleading sense of control over the software coding activities to follow.

  4. The supposed boundaries between requirements, design and coding are entirely porous, with all three “phases” of development interacting with and influencing the others.

  5. Prioritizing requirements is generally difficult, since every requirement is important to someone, and no one wants to call another’s child ugly.

  6. The cost of making software changes has consistently declined, due to the increasing power and sophistication of the languages and tools available to modern developers. And so writing software and then changing it based on reactions from stakeholders is often more efficient than going through elaborate and unreliable paper-based rituals prior to coding.

  7. Any software development effort is either a journey of discovery, or a waste of resources. Either you are creating something new, or you are recreating something that’s already been done. If you are creating something new, then you had best be honest that you aren’t entirely sure what you’re doing, or how long it will take, until you’ve done it. If you’re entirely sure of what you’re going to do and when you’re going to do it, then you’re not developing something, you’re just putting a fresh coat of paint on something that already existed.

  8. All software applications are built on top of hardware and software platforms that are themselves highly complex and still rapidly evolving. There are no limitations. There are no hard and fast rules. Everything is malleable, and everything is possible. In this sort of environment, it is folly to try to define some sort of repeatable and stable engineering discipline.

  9. The demonstration of working software is the only reliable indicator of progress in any sort of software development activity. All other avowals of project progress are illusory.

  10. Customers are happiest when the software is adaptive and evolving quickly in response to their explicit and implicit wants and needs.

Of course, for many of us, none of this comes as a surprise. After all, all of this is why agile software development has emerged and attained such dominant mindshare over the last decade or so.

So why do I feel the need to restate all of this now? Perhaps it is because, despite the apparent popularity of agile practices, the old myths of predictive development still prove so convenient and comforting. Like beliefs in intelligent design, or in Newtonian physics, they somehow just feel so much more natural and comfortable than their more modern counterparts.

And so it is tempting from time to time to slip into these old beliefs, and to pretend that we can have our cake and eat it too, be all modern and agile but retain some vestiges of this old predictive waterfall model.

So perhaps it is important, not just to state our support for agile, but to go through this catechism, to cast out the waterfall devils, to exorcise these ghosts one more time, to remind ourselves that, no matter how convenient and comforting these old beliefs may be, they simply are not true.

October 11, 2013

Next: This Thinking Business