«    »

Problems with Typical Definitions of Project Success

My previous article discussed the importance of defining success as it relates to software projects and products. Now I want to look at some typical definitions of success and identify problems or short-comings with these definitions. From this analysis I aim to point the way towards a better definition of what success entails.

Typical Definitions of Success

What are common definitions of success?

The fourth edition of the Project Management Body of Knowledge (PMBOK) frequently refers to project success, but does not appear to give a concrete definition which I find a little unsettling. All it seems to state on the subject is that the project charter should define the success criteria / objectives.

The Standish Group is well known for publishing their Chaos report that identifies leading causes of I.T. project failure. Success is defined as a software project that finishes on time, on budget, and within scope (e.g. with no missing functionality).

The article http://www.it-cortex.com/Stat_Failure_Rate.htm provides a number of different assessements of I.T. project success including the Standish group`s Chaos report that generally involve time, budget, and scope.

Formal definitions of success may seem too theoretical. Perhaps more relevant is how are projects measured and assessed within the organization, both in terms of the project status while it is in progress, and the evaluation of project success after the conclusion of the project. While I cannot speak for your organization, the ones I have worked for, observed, or heard case studies about all measure time and budget, and usually measure scope.

Many definitions of software project success that I was able to find generally correspond to the following definition: "A high-quality product that satisfies agreed requirements within time and budget" taken from the book A Practical Approach to Software Quality by Gerard O'Reagan on page 215. For the remainder of this article I refer to this as the typical definition of success.

Success and Failure Scenarios

The typical definition of success may be quite popular and common, but is it useful, accurate, or comprehensive? To put it to the test I present a number of project scenarios – some real, some hypothetical – and see whether the typical definition of success matches what the situation suggests.

Death March

The project schedule and/or scope was overly aggressive. The project team worked long hours for an extended period of time in order to get the software completed within budget and schedule. A few team members quit or burnt out during the project, and most resigned shortly after the project finished.

Strictly speaking this is a hypothetical scenario, but I have heard of a few projects at various game companies that have come close. The book Death March by Edward Yourdon would not have become a classic if this scenario was not a common occurance within the industry.

Analysis: While the typical definition would call this project a success, I consider it a failure, and I would hope most reasonable organizations would as well. Definitions of success need to consider the project's impact on the team.

Unsatisfied Users

The project completed within schedule and budget with slightly more functionality than the original scope. But the users felt that their requirements had been misunderstood by the project team and that the software did not meet their needs.

This scenario is based on a project I was involved with a few years ago.

Analysis: My initial reaction at hearing of unsatisfied users is to label the project a failure, even though it qualifies as a success according to the typical definition. Definitions of success should consider the satisfaction of the customer and end users of the software.

However, given my experience with this scenario, I know that an important question to ask is whether the users' dissatisfaction was reasonable- i.e. did they know what they wanted, and was it achievable? I am reminded of a Dilbert cartoon where Dilbert is gathering requiremnts from a marketing person who starts fantasizing about a telepathic user interface. Some people you cannot make happy, no matter how hard you try.

Marketplace Failure

Company executives considered the project a success because it had been run by the book and the final system was very similar to the initial specification. But an external expert who evaluated the project found that it scored poorly in quality and productivity and faired poorly in the marketplace. Ironically, a second project that company executives considered a failure because the project was continually changing in efforts to respond to changes in the marketplace ended up quite successful in the marketplace.

This scenario is described in page 30 of the book Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck and comes from a paper by Alan MacCormack, a professor at Harvard who was the outside expert.

Analysis: There are several lessons here. The first is that definitions of success should consider how the product performs in the marketplace. This relates to the prior point regarding satisfaction of customers / end users - the more satisfied the consumer is, the more likely the product will excell in the marketplace. One question to consider, however, is whether the success of the project should be completely dependent on the success of the software that is produced. One can easily imagine scenarios where a team does a great job producing a great product, but it fails in the marketplace for reasons outside their control (e.g. bad marketing).

The second lesson is to observe the danger of having a wrong definition of success. Those company executives, who presumably make decisions to approve and fund projects and reward project teams, evaluated project success completely opposite to the level of success in the marketplace. The risk is that these executives cancel the 'bad' projects and fund more projects like the 'good' one, and put their company out of business as a result.

Late but Leader

The project was years late before it was finally delivered: at no point in time was the project estimated to take more than about one year, but it actually took just over five years to ship. The software, however, became number one in the marketplace and a flagship product for the company.

This story is told in the book Rapid Development by Steve McConnell (pages 207 - 208) about Microsoft Word for Windows version 1.0. I assume you have heard of this piece of software :)

Analysis: This scenario is essentially the opposite of the prior scenario and thus raises the same question about whether product success means project success. In this case, given how badly the project was estimated and the negative impact that the overly-aggressive schedule had on the team's efforts, I would be hard-pressed to call the project itself a success, although the product itself was extremely successful.

Stopped for Higher Priority

The project was stopped by management part-way through with only partial functionality delivered in order for the team to start a new, much higher priority project to take advantage of an excellent opportunity in the marketplace.

This scenario is hypothetical.

Analysis: The typical definition of success would call this project a failure. But why, exactly? Because not all the functionality was delivered? If management determined that continuing to work on the remaining functionality was less valuable than starting the new project, then wouldn't the really failure have been to continue with the project (assuming no other people were available to start the new project)? When a project is started, certain decisions are made regarding its scope, schedule, and budget, usually based on limited information. If these decisions are changed later on the basis of more accurate information, how is this a failure? Another point I've hinted at here is that worthwhile projects must be delivering value to some customer or stakeholder, and defintiions of success should consider this.

Cancelled

The project was cancelled after a few months after it became clear that the new leading-edge technology being piloted was not working out and that the project's expected return on investment would not be achieved.

This scenario is hypothetical.

Analysis: This continues the theme of the last scenario regarding the delivery of value. Projects that are not delivering value should be stopped (or fixed). But is a cancelled project a failure? You might be tempted to say yes. I was. But consider that the typical definition of success considers cancelled projects failures, whereas if such a project was left running to its conclusion it would be considered a success, irrespective of the value produced. People try to avoid failure, so calling this scenario a failure motivates people to avoid it and continue down the path of not providing value. But is it right to call this project a success? One could argue that it did provide value in terms of the organization learning that the new technology was unworkable. If evaluating the technology was one of the primary goals of the project, then that goal was definitely achieved.

Simultaneous Success and Failure

The project was considered a failure by management due to not meeting its estimates (it was 400% over budget), but a majority of the developers considered it their most successful project.

This scenario is told in the book Facts and Fallacies of Software Engineering by Robert Glass, page 39, and comes from research performed by K.R. Linberg.

Analysis: Robert Glass's analysis of this situation identifies that one of the underlying factors behind this disagreement is an "unspoken controversy ... about what constitutes project success." (page 41). My view is that different stakeholders can hold widely differing views of success. (And yes, I consider the project team to be a significant stakeholder in the project - they are the ones dedicating a portion of their lives to it.) Definitions of success should take into consideration these different views, but how do you reconcile them when they conflict? Is one stakeholder's view better or more correct than anothers? This goes back to my prior discussion on customer and user satisfaction, and what happens if they are unreasonable. The same can be said of management's expectations - if the initial schedule or budget is completely unrealistic (which the developers in this scenario did identify as a reason the project was late), then why shouldn't the management view of success be disregarded?

No Budget or No Schedule

Some projects, for all intents and purposes, have no budget or no schedule. The best example is open source projects. Some of bigger open source projects do have a target date for their next release, but keep the actual release date flexible based on when the software is ready. And virtually all open source projects have no budget at all.

Analysis: The typical definition of success is unable to even evaluate the success or failure of such projects, which seems wrong for a supposedly universal definition. You might try to claim that open source development does not consitute a project, but any given release of open source software has a clearly defined start and end, with activities being performed within that duration to achieve the goal of shipping that release. That sounds like a project to me. How would you judge the success of open source development if there is no budget, a fluid schedule (maybe), and maybe not even a defined scope? I think the answer lies in determining whether the stakeholders are satisfied and whether value is being produced for these stakeholders.

Conclusion

I purposely selected scenarios that challenge the typical definition of sucesss in order to highlight its flaws and to identify considerations that can lead towards a better definition of success. The essential underlying problem, I believe, is a broken paradigm, which I illustrate by rewording the typical definition of success in story form:

When we are about to start a project is when we know the least about it, which means that our estimates will be the worst possible. Let us guess, up front, precisely the scope (functionality) we want to deliver and then guess the budget and schedule required to deliver that scope. These guesses may then be arbitrarily revised downward for budget and schedule and upward for scope for political reasons. Let us then commit to these guesses. If we do not achieve them our project will be judged a failure, no matter what we might learn throughout the project.

Hopefully worded in that fashion it is clear that there is something fundamentally disfunctional about evaluating projects in this manner. My next article in this series on success will focus on coming up with a better definition of success.

P.S. I must point out that I am being somewhat unfair in using the term "typical definition of success" as others have had similar thoughts to myself and have come forward with alternate definitions. See for example Scott Ambler's analysis.

If you find this article helpful, please make a donation.

«    »