While the term agile itself is indeed ten years old, the philosophy and approach of iterative and incremental development (IID) has a surprisingly rich and extensive history. The article Iterative and Incremental Development: A Brief History (PDF) by Craig Larman and Victor R. Basili published in IEEE Computer discusses how the ideas of IID actually predate the existence of software, and have been used and promoted in every decade since for over 50 years. This article also discusses a classic 1970 article by Winston Royce well-known for supposedly promoting the use of waterfall, but which actually recommended the development of an initial pilot or preliminary version prior to creating the final version intended for delivery to the client. Larman's and Basili's article also discusses the ongoing evolution of the standard used by the U.S. Department of Defense for software development. The initial standard was document-heavy, gated, single-pass waterfall, but the high rate of project failures led to first the allowance of and eventually the full adoption of IID approaches.

So a limited set of organizations (often governments due to their byzantine contracting restrictions) did experience an evolution from waterfall to agile over time, but notably they started with a misunderstood version of waterfall. The use of IID approaches has always been part of the I.T. industry.

What about the effectiveness of agile? The second article Quality metrics: Software quality attributes and their rankings is an interview with Capers Jones and Olivier Bonsignour regarding their new book The Economics of Software Quality. The authors in this book discuss the effectiveness of over 100 quality factors using research based on over 10,000 software projects. On a scale of -10 for extremely harmful to a scale of +10 for extremely valuable, Agile methods rated a 9 - highly valuable - while waterfall only rated a 1 - barely useful.

These two articles highlight the fact that agile-like methods have been in use since the start of the software field, and are on average far more effective than the waterfall approach. This leads me to conclude that there is really no defensible reason for organizations to mandate or promote the use of waterfall.

Dealing with Uncertainty

Mike characterized traditional approaches as trying to manage out uncertainty in the effort to achieve predictable results. This strategy essentially involves determining up front the key aspects of the project (scope, budget, and schedule) and then seeking to minimize changes to it throughout the project. Unfortunately, virtually all software development is faced with a multitude of uncertainties. Trying to drive them out means conflicting with reality, which leads to a common litany of project problems such as: over budget, over schedule, scope achieved but business not satisfied (business objectives not met), and excessive cost for realized value.

Agile adopts the opposite philosophy: manage for uncertainty. Kent Beck used the term "embrace change" when first introducing Extreme Programming, and this is a common goal of all Agile methods. This is achieved through several strategies:

• Requirements are defined and implemented throughout the project, making it trivial to accommodate changes. The next section below discusses this in more detail.
• Providing frequent feedback via working software mitigates many of the risks of traditional development projects.
• Prioritizing user stories and first implementing the ones with the highest value / highest risk maximizes the return on investment and mitigates a number of common risks.
• Often Agile projects explicitly allow for the project to be stopped or brought to a close at any point if the business owner determines that there is no benefit in proceeding with the remaining requirements.

One of Mike's though-provoking points was that using an Agile method is a risk-mitigation strategy that mitigates many of the common risks associated with software development projects.

Deriving instead of Fixing Scope

One key aspect of project management is managing the iron triangle of scope, cost (budget), and time (schedule). In a traditional waterfall project the scope is defined first, the effort required to implement this scope is then estimated, and finally the budget and time required are derived from this estimate. This approach is maintained throughout the project: controls are often put in place to minimize changes to the scope and changes that are deemed necessary then typically require adjustments to the budget and/or schedule.

Fixing the scope up front is inherently flawed due to the nature of software development:

• Business requirements almost always change over time. I have seen statistics mentioned on the order of 1% change in requirements per month.
• Software development is inherently a learning activity. The business typically gains clarity about their requirements as they are analyzed and implemented. Insight is also gained into how the solution / system can best meet these requirements.
• One technique from lean software development is set-based concurrent engineering, which aims to defer decisions to the last possible moment when the most information is available. This helps ensure that the best possible decision is made and hence minimizes waste caused by bad decisions (one of the main objectives of lean). Fixing scope up front requires making decisions at the worst possible time – when the least is known.

Agile addresses this problem by making scope flexible. While an Agile project can (and usually does) start with a high-level backlog of features or epics, the specific user stories selected for implementation are determined throughout the project, iteration by iteration. I have heard some say that Agile makes the project scope easy to change, but in reality the scope is not being changed at all – it is being defined throughout the project. The total scope achieved in a project is hence a function of the project schedule: the more iterations and releases available, the more that can be done. Going back to the triangle of scope, cost, and time, this means that time is the main variable being controlled, from which budget and scope can be derived.

Conclusion

I really enjoyed the clarity of thinking Mike brought to the topic of Agile project management. Considering that Mike has given this presentation at major software development conferences that require $$$ to attend, I really appreciated having the opportunity to see it for free in my home town. I am a little surprised that there are not more people taking advantage of these opportunities. If you live in the Edmonton area I strongly encourage you to attend the Agile Edmonton meetings.

But then I read an interview with Alastair Cockburn in which he talked about what it means to be agile. Cockburn provided a top ten list of indicators that you are not agile. As I read this list, I realized with a sinking feeling that most of these applied to my current team. I went back through the list and recorded my team's score out of 10 for the indicators for which we were agile. I even gave us half a point for those items for which we were partially agile. Our final score was a measly 3 out of 10 (10 being fully agile). Clearly we are not agile.

Would I have to revoke my agile membership? I still believed in the agile approach, so how could I not be agile? Or was I mistaken? I re-read the Agile Manifesto, and still agreed with what it said. What about the principles of agile software? I re-read those as well, and again found myself in agreement. There were a few principles that are difficult or impossible to apply in my current environment, but that has not stopped me from trying to do what I can.

I agree with the agile values and principles and try to apply them within my work environment. I believe this is the fundamental indicator of whether you as an individual are agile or not, rather than a list of specific practices. Therefore I am agile.

A separate question is whether I, as part of a team, am doing agile software development. Despite the flaws with Cockburn's top ten list, it does help evaluate this second question. The answer for myself is no. I am an agile software developer doing non-agile software development. The distinction may be subtle, but it is significant. To help you see it, imagine the opposite case of a non-agile developer working on an agile software development process. They may be following the team's agile processes while pushing for more formal documentation and more detailed processes. Your team's approach does not necessarily reflect the beliefs and values of each team member. Another way of viewing this distinction is that agile can be thought of as both a philosophy consisting of a set of values and principles, and as a set of practices which implement and uphold these principles and values. I am unable to follow all of the practices at work, but I still subscribe to and apply the values and principles.