As I often do, I found myself reflecting at the end of these days as to whether I had properly prioritized my time. I could have been resolving tricky technical issues and advancing progress towards our desired future state in terms of architecture, technology, and development practices. But I am only one person. I realized that the focus on growing our group of solid Java developers and architects would pay off in the long run by increasing the overall capacity of problem-solving and progress-advancing.

Another consideration is that the implementation of the improvements that I wanted to see happen are limited by the capabilities and knowledge of the developers who actually put these things into practice. In the past, I have experienced problematic developers who inadvertently introduce issues that others are stuck resolving and use older technologies and/or less-effective practices through lack of awareness or poor habits. Hiring highly capable developers, in contrast, makes it much easier to make progress on these improvements. So for a limited short-term investment of my time, focusing on hiring has staggeringly immense long-term benefits.

The above logic regarding my personal limits apply just as equally to my direct involvement with hiring. As much as I may like to personally interview each applicant, I simply do not have the time, especially during hiring booms. So my objective in working on the hiring process is to put a system in place that can, if need be, operate without my involvement while obtaining the same results.

I received feedback recently from one director that the new hires we have brought on since my involvement in hiring have been almost entirely working out just fine. This is very rewarding to hear and is confirmation that my efforts in building an effective hiring system have indeed paid off.

What is it about the spreadsheet that is so alluring? There is some value to studying it - I think of it as uploading the numbers into my subconscious to get an intuitive feel for where there might be obstacles or challenges that the team is facing now or will face shortly in the future. But the value of doing so quickly drops off and yet we keep staring. Part of the temptation I feel is to find ways to bump that daily or weekly burndown just a little higher by finding that task that is actually complete or is over-estimated. This is especially true if progress has not been good.

So what can we do to stop starting at the spreadsheet?

1. Remember that identifying obstacles is no good unless they are dealt with. Focus on removing or mitigating the obstacles instead.
2. Ask yourself what activity you can do that will best accelerate the team's progress. This might be helping a developer that is stuck on a technical issue, freeing up the time from a tester who has become a bottleneck, or clarifying requirements for upcoming work. More studying of the spreadsheet is never the answer. This is especially useful when the team's progress has been lower than expected.
3. Remember that minor errors in the spreadsheet are inconsequential in the grand scheme of things. Forgetting to burn down a task or two is dwarfed by the innate inaccuracies of the estimates you are using to track progress. If you miss updating a task this week, you will likely catch it next week.
4. Remind yourself that the spreadsheet is merely a tool, and that you are the master of the tool - not the other way around.
5. Throw away the spreadsheet and use a simpler approach - perhaps a paper burndown chart on the team wall. Just ensure you do not then start staring at the wall.

In one prior project as we approached the planned deadline for starting user acceptance testing (UAT), I warned the project manager that the code quality was poor and only limited testing had been done so UAT would likely be a disaster. My advice was to miss the deadline and ensure good quality prior to entering UAT. Unfortunately, the manager decided the milestone was more important than getting the software to a working state. The code went into UAT and was soon thereafter rejected as being too buggy and unstable.

On another project a number of factors including excessive bureaucracy and externally-driven requirement changes caused delay after delay. After a few years (yes, multiple years) of delays, everyone from the project sponsor down to the project team members were so tired of the project dragging on that scope was massively cut and it was essentially rammed into production in a mostly non-functional shape, leaving the maintenance team to pick up the pieces.

Instead of focusing primarily on the plan, I propose to focus primarily on the goal - the real objectives. Progress should be evaluated as much as possible in terms of achieving objectives. I expect a common objection from plan-driven proponents would be goals are at too high a level for tracking progress. The solution is to decompose goals into the smallest possible objectives that retain meaning. In software development this means decomposing requirements or features into the smallest possible size that is still business-meaningful. Ron Jeffries calls this the minimum marketable feature (MMF) and views the delivery of working software that achieves MMFs as the fundamental unit of progress for software development.

Focusing on the goal means that a change in objectives should result in a change in action as soon as possible. Plan-driven methods seek to limit change, often explicitly, in order to stay aligned with the plan. This leads to misalignment with the goal, which over time leads to unhappy customers/users or effort wasted on pursuing the wrong direction. Most software development projects involve continual learning by both the business and by the developers about the problem domain, the solution domain, and the technical implementation of the solution. This inherently leads to objectives changing over time.

I do believe in planning, but it must remain subordinate to the objectives. The plan must remain flexible and easy to adjust based on how the objectives are being met and how they are evolving over time. Dwight D. Eisenhower is reported to have said "Plans are worthless, but planning is everything." This quote captures the essence of my philosophy. The act of planning is critically important - determining the correct course of action based on current understanding - but plans based on old and likely outdated knowledge are not.

Here are ten specific warning signs you can look for, expressed as questions to ask of yourself, your management, and your organization as a whole.

1. Given the choice between delivering service requested by a customer in variance with a defined process versus following the process and leaving the customer unhappy, what is the normal choice?
2. How would management react to the deviation from process above?
3. Can processes be easily changed when improvements are proposed, or is this difficult due to the need to pass audits involving the process.
4. Is the focus of most process improvement on addressing audit non-conformities and passing the next audit, or is the focus on improving business-meaningful objectives?
5. What percentage of management activities such as communicating in team meetings or providing direction to subordinates consists of instructing people to follow processes with the primary rationale being to pass audits.
6. Does management provide special directions to subordinates immediately prior to an audit?
7. Immediately prior to an audit, do teams spend time ensuring documented evidence is up-to-date, or worse forge it, in order to pass the audit?
8. Are teams routinely producing documentation solely for the sake of producing audit-able evidence?
9. Do teams regularly complain about certain portions of an audited process? Do they provide feedback that they do not see the point of a specific portion of the process (or all of it)?
10. Do you find yourself frustrated with a particular audited process? Do you find yourself daydreaming about how to change it or eliminate it?

To those with a bureaucratic mindset process and audits are the answer to every problem: if something goes wrong, add more process and then audit to ensure it is followed.

Audits serve a purpose, but they do have drawbacks. Over-reliance on audits can actually cause negative consequences to the organization which are often not taken into account by those pushing for more auditing. Audits should be designed to minimize these drawbacks, and the organization should introduce additional mitigations as necessary. In fact, I would go as far as saying that the use of audits should be kept to a minimum.

So what are these drawbacks? I have organized them into the following categories which are explored in detail in the remainder of this article.

• Reduces Productivity
• Harms Organizational Culture
• Limits of Assessment

Reduces Productivity

I define productivity as the amount of value produced for stakeholders based on the effort expended. In other industries such as manufacturing with more concrete notions of value it is possible to effectively measure productivity. In I.T., however, this is difficult to do, and I have never heard of an I.T. audit that evaluates productivity, value produced, or even effort expended. Audits typically instead measure compliance to documented processes based on historical documentation regarding the work that was done. This disconnect inevitably leads to a reduction in productivity in the following ways:

Do work to pass audit rather than deliver value

Auditing documented evidence of following documented processes causes people to produce documentation, whether or not it provides value, in order to pass the audit. This is especially true when the work becomes about producing the documentation instead of producing value.

Assessing quality after the fact is wasteful

Two important concepts from lean thinking are that any time delay in a process is waste, and that quality should be built into the process to avoid the waste associated with undetected quality problems remaining in the work-in-progress. Performing an audit long after an activity has been performed violates both of these concepts. In some cases I have seen audits are performed months after the activity has been completed - for example, an audit of a software enhancement that has already been deployed to production. If problems are found, it is too late for that activity. And the follow-up to determine if the problem has been resolved typically does not happen until the next audit, which is an even longer delay.

Harms Organizational Culture

An organization in which formal audits are a common occurrence risks, in my opinion, serious harm to the organizational culture by shifting the culture towards a more bureaucratic, stagnant, and confrontational atmosphere. This can happen in a number of ways:

Shift of focus away from organizational objectives

People generally have difficulty focusing on more than one or two priorities at a time. When the focus is placed on passing audits, attention is diverted away from achieving the organization's objectives or purpose. This is similar to the problem faced by traditional project management - making scope, schedule, and budget the goal runs the risk of not actually meeting the business needs.

Compliance mindset rather than improvement mindset

Formal audits, especially when they are treated strictly by management, tends to lead to a fear-based compliance mindset. The unspoken message is "pass the audit, or else...". This is inimical to a mindset of continuous improvement, specifically the attitude of being willing to change the process when a better approach is found. In theory a process-centric mindset and a continuous improvement mindset can coexist - Toyota seems to be successful at this. But adding audits into the mix adds a rigidity or bureaucracy to processes that impedes continuous improvement. Worse is when people blindly follow process rather than question whether there is a better way.

Attitude of confrontation rather than collaboration

Formal audits almost always involve separate auditors external to the group being audited, and sometimes external to the organization. And these auditors are trained and motivated to find non-conformities - deviations from process. Auditors are not engaged to work with the group to understand what they have done and to help them do better. In fact, they are often instructed to remain aloof in order to remain impartial. This tends to lead to an attitude of confrontation rather than collaboration - an us-versus-them mentality - which means that any useful feedback that the auditors might have is at risk of being denied or ignored.

Discourages context-based expert judgement, creativity, and initiative

The Dreyfus model of skill acquisition explains the difference between beginners and experts. Having a clearly defined, step-by-step process to follow is essential for beginners, but has been demonstrated to actually reduce the performance of experts, who use their judgement based on the context at hand to determine what to do and how to do it.
Enforcing adherence to a common process also discourages creativity, initiative, and innovation. This can happen in a bureaucratic environment without audits, of course, but I believe that a regime of formal audits will exasperate the problem.

Limits of Assessment

Audits at their essence are about assessing a group with respect to some standard. This assessment has certain fundamental limitations due to the nature of audits, and over-reliance on audits risks developing an incomplete or inaccurate picture of reality. These limitations include:

Cannot assess intangibles

Audits only assess concrete items that are documented and completely ignore intangible aspects of work such as motivation, creativity, and initiative. (Some of these intangibles can be assessed by other 'formal' mechanisms - e.g. assessing motivation through surveys.) The corollary to the adage "you get what you measure" is that you risk not getting what you don't measure. This would not be a problem if the intangibles were not important. But software development is knowledge work, and the research is clear that people are by far the most significant factor determining productivity, far ahead of process. So relying on audits, which typically only look at process, ignores the more intangible, people factors.

Assessment is twice removed from objectives

Each group or organization has a purpose for existence, and objectives by which it tries to achieve that purpose. Audits do not assess this. Organizations put processes into place to support meeting these objectives. But audits, strictly speaking, do not assess this either. Audits assess the historical, documented evidence that the processes were followed. This means they are twice removed from the true objectives, which makes them an incredibly weak tool for helping an organization meet its objectives.

Audit results may be invalid

Audits suffer from four failure modes which can cause them to report invalid results:

1. The group is actually following the process being audited, but is not correctly producing the evidence.
2. The group deliberately deviates from the process in order to meet organizational objectives or in order to be more effective.
3. The group produces the documentation used as evidence for following the process without properly following the process. They will pass the audit, but there is a hidden problem. Rubber-stamping, where a review or approval is given without any actual check or thought, is an example of this.
4. The group properly follows the process and produces the evidence, but fails to achieve organizational objectives.

In conclusion, be very careful in how you use audits and watch out for the drawbacks. Maybe you should audit your use of audits :)

Achieving the full definition of done for a feature or release (often called being done-done) is surprisingly difficult. You may have heard the anecdotal rule of thumb that it takes 80% of the time to achieve 80% complete on a feature, and another 80% time-wise to achieve the remaining 20%. This is not due to using a strict definition of done. Anyone who has worked on an undisciplined waterfall project that has 'finished' coding the functionality and is now in test knows that sinking feeling of having defect after defect reported with no end in sight – and hence no predictability.

Using an imprecise or overly-simplistic definition of done will definitely impede predictability because you have only a vague idea of what the current state of your software is, and hence have high uncertainty regarding your progress to date.

Using a specific definition of done combined with the lean principle of minimizing work-in-progress greatly improves your knowledge of the current state of your software and hence your progress. Most of the uncertainty has been driven out by forcing work to done-done. This is not enough, however, to improve the predictability of your budget or schedule estimates for your outstanding work. You need to ensure your planned tasks and estimates include everything required to get to done-done.

On my last project, we tried just tracking coding tasks in the sprint plan. The remaining effort to get to done-done was treated as part of the general overhead that was used to calculate our velocity. This provided poor predictability in terms of both budget and schedule, especially as a feature neared code-complete. The estimate on the remaining coding tasks approached zero, but the time required for reviews, testing, and fixes remained roughly constant, thus exceeding what could be accounted for by velocity.

We then started to track most checklist items from our definition of done as individual tasks for each feature in addition to the regular coding tasks provided by the developer(s). This worked better, but we still suffered predictability problems as the feature went through review and test. Defects discovered had to be fixed, impacting the schedule and budget for the feature.

The next tweak we made was to allocate time up front for these fixes by adding an entry titled "Anticipated Issues" to each feature that provided an allocation of effort (budget, from which schedule was derived) that could be used to address defects, review corrections, poorly estimated tasks, and discovered work. The amount of effort allocated to this item was estimated per feature based on the size and complexity of the feature and the experience and quality level of the developers working on the feature.

Our most recent adjustment was to the front-end of feature development. Over several sprints, we noticed a trend that when developers started a new feature, especially one that was significantly new, they often had a day or even two where they did not make the expected progress on their tasks. We eventually determined that developers consistently had to spend time familiarizing themselves with the feature, especially learning the requirements and getting clarifications regarding how the feature was to be implemented, and that they virtually never allocated time in their tasks / estimates for this. So we added a new task to each feature named "Feature Clarification" to make this explicit.

By planning for all the components of our definition of done, we have achieved much better predictability. While our variance from our estimates per feature remained fairly high, the average variance across features was much closer to the original estimate, rather than being consistently over. As progress is made on a feature, we have much better estimates and better visibility into the outstanding work.

One solution to this problem that I have worked with and now recommend is to use a feature done checklist to track the completion of the definition of done for a specific feature. At its essence the checklist consists of the major elements of the definition of done listed as rows in a table with additional columns for the person verifying the completion of the item to enter their name and date. Here is a simple example:

You need to decide whether to use digital or printed checklists. I prefer using paper-based checklists for several reasons:

• I find it more meaningful to sign a piece of paper than to add my name electronically, so I tend to treat the completion of the checklist more seriously than I would an electronic form. This attitude may be due to my engineering background.
• A paper copy is more tangible and real. It can be waved around at a team meeting to celebrate the completion of a feature or posted on a team wall. It can be placed on a coworker’s keyboard in order to focus their attention on an outstanding item.

Evolving the Checklist

I have found it useful on occasion to evolve the basic feature done checklist beyond the team’s definition of done to provide support for the team’s development process. This is usually done as part of continuous improvement to resolve issues identified during retrospectives. Here are some examples:

• On one project we had issues with developers rushing into coding new features without fully understanding the requirements, so we added an initial checklist item for feature clarification to focus attention on the activity.
• For each item you can specify the role (e.g. developer, tester, architect) responsible for signing that it is done. I strongly recommend doing this for two reasons: it minimizes confusion over who signs off each item, and it minimizes the problem of having the developer of the feature quickly sign off all the checklist items without doing the necessary checking so they can say the feature is done.
• You can specify dependencies between the items. On one project we had a problem with people treating the items as a linear sequence of activities, so we defined dependencies to explicitly show that some activities could be done in parallel.
• You can add a feature start date to indicate when work began on the feature. This in combination with the feature done date allows you to calculate the duration of time spent on the feature. In lean terminology, this gives you the cycle time for the feature, which you want to minimize. When aggregated across the team, this defines the team’s overall throughput in completing features.

Checklist Challenges

I have encountered a number of challenges when using feature done checklists. Some of these issues are due to the actual checklists, but many are actually due to using a strict definition of done – the checklists just make the issue visible.

1. The checklist needs to be meaningful to the team - you must avoid the risk of it just becoming a form that people sign without doing the necessary work / checking that the item entails. (This is sometimes called rubber-stamping or pencil-whipping.) I have used many strategies to mitigate this risk:
   • Use a paper-based checklist rather than electronic.
   • Keep the checklist to a single page with as few items as possible for each role to complete.
   • For items involving ‘checking’ activities, assign these to roles / people whose primary responsibility is checking, separate from the developers doing the coding.
2. Extra effort is involved in getting a feature fully complete. This is often not accounted for in estimates, particularly if the estimate is provided by the developer. The time required for checking by other roles can be accommodated by having separate estimated tasks for these activities. What I find particularly challenging is accounting for extra time needed by the developer to resolve issues found in reviews or testing.
3. Having a through definition of done with multiple hand-offs of the feature to various roles will typically extend the duration of time required to complete the feature. The mitigation is to allow as many items as possible to be done concurrently. The best example is using pair programming as the means for doing the peer code review.
4. A single role / person with multiple review items per feature can end up becoming a bottleneck that ends up delaying feature completion. This is especially problematic if you are using a methodology like Scrum which focuses on getting features (user stories) done within the iteration – you tend to get a backlog of checklist items forming at the bottleneck near the end of the iteration which jeopardizes their completion.
5. One problem with using paper-based checklists is that they can get misplaced, or you end up looking for who has a form. This can be addressed by defining a location for checklists to be returned to. One option I like is to post them on a team wall, which provides the additional benefit of visibility.

Typical Definitions of Success

What are common definitions of success?

The fourth edition of the Project Management Body of Knowledge (PMBOK) frequently refers to project success, but does not appear to give a concrete definition which I find a little unsettling. All it seems to state on the subject is that the project charter should define the success criteria / objectives.

The Standish Group is well known for publishing their Chaos report that identifies leading causes of I.T. project failure. Success is defined as a software project that finishes on time, on budget, and within scope (e.g. with no missing functionality).

The article http://www.it-cortex.com/Stat_Failure_Rate.htm provides a number of different assessements of I.T. project success including the Standish group`s Chaos report that generally involve time, budget, and scope.

Formal definitions of success may seem too theoretical. Perhaps more relevant is how are projects measured and assessed within the organization, both in terms of the project status while it is in progress, and the evaluation of project success after the conclusion of the project. While I cannot speak for your organization, the ones I have worked for, observed, or heard case studies about all measure time and budget, and usually measure scope.

Many definitions of software project success that I was able to find generally correspond to the following definition: "A high-quality product that satisfies agreed requirements within time and budget" taken from the book A Practical Approach to Software Quality by Gerard O'Reagan on page 215. For the remainder of this article I refer to this as the typical definition of success.

Success and Failure Scenarios

The typical definition of success may be quite popular and common, but is it useful, accurate, or comprehensive? To put it to the test I present a number of project scenarios – some real, some hypothetical – and see whether the typical definition of success matches what the situation suggests.

Death March

The project schedule and/or scope was overly aggressive. The project team worked long hours for an extended period of time in order to get the software completed within budget and schedule. A few team members quit or burnt out during the project, and most resigned shortly after the project finished.

Strictly speaking this is a hypothetical scenario, but I have heard of a few projects at various game companies that have come close. The book Death March by Edward Yourdon would not have become a classic if this scenario was not a common occurance within the industry.

Analysis: While the typical definition would call this project a success, I consider it a failure, and I would hope most reasonable organizations would as well. Definitions of success need to consider the project's impact on the team.

Unsatisfied Users

The project completed within schedule and budget with slightly more functionality than the original scope. But the users felt that their requirements had been misunderstood by the project team and that the software did not meet their needs.

This scenario is based on a project I was involved with a few years ago.

Analysis: My initial reaction at hearing of unsatisfied users is to label the project a failure, even though it qualifies as a success according to the typical definition. Definitions of success should consider the satisfaction of the customer and end users of the software.

However, given my experience with this scenario, I know that an important question to ask is whether the users' dissatisfaction was reasonable- i.e. did they know what they wanted, and was it achievable? I am reminded of a Dilbert cartoon where Dilbert is gathering requiremnts from a marketing person who starts fantasizing about a telepathic user interface. Some people you cannot make happy, no matter how hard you try.

Marketplace Failure

Company executives considered the project a success because it had been run by the book and the final system was very similar to the initial specification. But an external expert who evaluated the project found that it scored poorly in quality and productivity and faired poorly in the marketplace. Ironically, a second project that company executives considered a failure because the project was continually changing in efforts to respond to changes in the marketplace ended up quite successful in the marketplace.

This scenario is described in page 30 of the book Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck and comes from a paper by Alan MacCormack, a professor at Harvard who was the outside expert.

Analysis: There are several lessons here. The first is that definitions of success should consider how the product performs in the marketplace. This relates to the prior point regarding satisfaction of customers / end users - the more satisfied the consumer is, the more likely the product will excell in the marketplace. One question to consider, however, is whether the success of the project should be completely dependent on the success of the software that is produced. One can easily imagine scenarios where a team does a great job producing a great product, but it fails in the marketplace for reasons outside their control (e.g. bad marketing).

The second lesson is to observe the danger of having a wrong definition of success. Those company executives, who presumably make decisions to approve and fund projects and reward project teams, evaluated project success completely opposite to the level of success in the marketplace. The risk is that these executives cancel the 'bad' projects and fund more projects like the 'good' one, and put their company out of business as a result.

Late but Leader

The project was years late before it was finally delivered: at no point in time was the project estimated to take more than about one year, but it actually took just over five years to ship. The software, however, became number one in the marketplace and a flagship product for the company.

This story is told in the book Rapid Development by Steve McConnell (pages 207 - 208) about Microsoft Word for Windows version 1.0. I assume you have heard of this piece of software :)

Analysis: This scenario is essentially the opposite of the prior scenario and thus raises the same question about whether product success means project success. In this case, given how badly the project was estimated and the negative impact that the overly-aggressive schedule had on the team's efforts, I would be hard-pressed to call the project itself a success, although the product itself was extremely successful.

Stopped for Higher Priority

The project was stopped by management part-way through with only partial functionality delivered in order for the team to start a new, much higher priority project to take advantage of an excellent opportunity in the marketplace.

This scenario is hypothetical.

Analysis: The typical definition of success would call this project a failure. But why, exactly? Because not all the functionality was delivered? If management determined that continuing to work on the remaining functionality was less valuable than starting the new project, then wouldn't the really failure have been to continue with the project (assuming no other people were available to start the new project)? When a project is started, certain decisions are made regarding its scope, schedule, and budget, usually based on limited information. If these decisions are changed later on the basis of more accurate information, how is this a failure? Another point I've hinted at here is that worthwhile projects must be delivering value to some customer or stakeholder, and defintiions of success should consider this.

Cancelled

The project was cancelled after a few months after it became clear that the new leading-edge technology being piloted was not working out and that the project's expected return on investment would not be achieved.

This scenario is hypothetical.

Analysis: This continues the theme of the last scenario regarding the delivery of value. Projects that are not delivering value should be stopped (or fixed). But is a cancelled project a failure? You might be tempted to say yes. I was. But consider that the typical definition of success considers cancelled projects failures, whereas if such a project was left running to its conclusion it would be considered a success, irrespective of the value produced. People try to avoid failure, so calling this scenario a failure motivates people to avoid it and continue down the path of not providing value. But is it right to call this project a success? One could argue that it did provide value in terms of the organization learning that the new technology was unworkable. If evaluating the technology was one of the primary goals of the project, then that goal was definitely achieved.

Simultaneous Success and Failure

The project was considered a failure by management due to not meeting its estimates (it was 400% over budget), but a majority of the developers considered it their most successful project.

This scenario is told in the book Facts and Fallacies of Software Engineering by Robert Glass, page 39, and comes from research performed by K.R. Linberg.

Analysis: Robert Glass's analysis of this situation identifies that one of the underlying factors behind this disagreement is an "unspoken controversy ... about what constitutes project success." (page 41). My view is that different stakeholders can hold widely differing views of success. (And yes, I consider the project team to be a significant stakeholder in the project - they are the ones dedicating a portion of their lives to it.) Definitions of success should take into consideration these different views, but how do you reconcile them when they conflict? Is one stakeholder's view better or more correct than anothers? This goes back to my prior discussion on customer and user satisfaction, and what happens if they are unreasonable. The same can be said of management's expectations - if the initial schedule or budget is completely unrealistic (which the developers in this scenario did identify as a reason the project was late), then why shouldn't the management view of success be disregarded?

No Budget or No Schedule

Some projects, for all intents and purposes, have no budget or no schedule. The best example is open source projects. Some of bigger open source projects do have a target date for their next release, but keep the actual release date flexible based on when the software is ready. And virtually all open source projects have no budget at all.

Analysis: The typical definition of success is unable to even evaluate the success or failure of such projects, which seems wrong for a supposedly universal definition. You might try to claim that open source development does not consitute a project, but any given release of open source software has a clearly defined start and end, with activities being performed within that duration to achieve the goal of shipping that release. That sounds like a project to me. How would you judge the success of open source development if there is no budget, a fluid schedule (maybe), and maybe not even a defined scope? I think the answer lies in determining whether the stakeholders are satisfied and whether value is being produced for these stakeholders.

Conclusion

I purposely selected scenarios that challenge the typical definition of sucesss in order to highlight its flaws and to identify considerations that can lead towards a better definition of success. The essential underlying problem, I believe, is a broken paradigm, which I illustrate by rewording the typical definition of success in story form:

When we are about to start a project is when we know the least about it, which means that our estimates will be the worst possible. Let us guess, up front, precisely the scope (functionality) we want to deliver and then guess the budget and schedule required to deliver that scope. These guesses may then be arbitrarily revised downward for budget and schedule and upward for scope for political reasons. Let us then commit to these guesses. If we do not achieve them our project will be judged a failure, no matter what we might learn throughout the project.

Hopefully worded in that fashion it is clear that there is something fundamentally disfunctional about evaluating projects in this manner. My next article in this series on success will focus on coming up with a better definition of success.

P.S. I must point out that I am being somewhat unfair in using the term "typical definition of success" as others have had similar thoughts to myself and have come forward with alternate definitions. See for example Scott Ambler's analysis.

Defining what makes for a successful project / product is critically important at every level: the individual, the team, the organization, and even the entire I.T. industry. (In the remainder of this article I will refer solely to projects with the understanding that the material applies as well to ongoing work on software products.)

Individual

Individual performance is often evaluated at least in part by the success or failure of the projects you work on, more so if you are in a leadership or management role. This can impact your compensation (raises, bonuses), future assignments and promotions, and even your job (if the project is a bad failure).

Being clear about the project’s definition of success should help guide you in the day-to-day decisions you make and actions you take on the project. This provides two benefits: you are more likely to be personally judged a success (even if the project fails), and the project is more likely to be successful.

Team

The choices to make in assembling a team – how many people will be on it, and what skill levels & abilities will they have – are significantly influenced by the project’s definition of success. For example, if minimal budget expenditure is a critical success factor then a small team should be used.

Once the team is assembled having a clear definition of success shared by the team members helps provide coordination of activity rather than disjointed efforts.

The practices & methods used by the team should be selected and tailored to maximize the likelihood and magnitude of success.

Organization

Within an organization that produces software decisions must be made regarding what projects to proceed or continue with and what level of funding and people to allocate to these projects. In a rational organization these decisions should be based on a cost-benefit analysis which should, in turn, be linked to the project’s definition of success. For example, if a project is conceived of to produce software targeted to hit the market in time for the Christmas selling season then hitting the delivery date becomes a significant factor of project success.

I.T. Industry

Our favorite pastime within the I.T. industry may well be promoting or debating the merits of various programming languages, environments, tools, practices, and methods. But what is the basis for making these evaluations? Presumably this is based on the likelihood of the tool, practice, or method making the project more successful (or less likely to fail). This evaluation should depend, therefore, on the definition of success being used, but this is seldom defined explicitly as part of the promotion / evaluation being done. (I am as guilty of this as everyone else.)

One notable exception is the book Rapid Development by Steve McConnell. Part III of the book lists 27 ‘best’ practices. For each practice a section on efficacy is provided that evaluates the practice based on three separate factors: potential reduction from nominal schedule, improvement in progress visibility, and effect on schedule risk. (Other project success factors such as quality or cost were omitted because the book focuses on minimizing schedule – thus the name Rapid Development.) McConnell explicitly advocates “selecting rapid-development practices that meet the needs of your specific project” (page 396).

Some might argue that virtually all projects share the same definition of success (usually along the lines of delivering required functionality with sufficient quality on time and on budget), and this then allows for meaningful discussion of 'best' practices independent of the definition of project success. I disagree with this position, but will save further discussion for a subsequent article: Problems with Typical Definitions of Project Success

Why is having a definition of done so valuable? I have grouped the benefits it provides into the following categories:

Higher Quality

Definitions of done should typically focus less on the activities people normally do and focus more on the activities that tend to be neglected or forgotten. In software development this means specifying activities that ensure the quality of the code being produced, rather than the actual act of coding. Quality often means much more than just few defects: it can also mean readable code, a clean design, or achieving any number of other non-functional requirements such as security, performance, and scalability. A definition of done that includes quality aspects serves, therefore, as both a reminder of the importance of quality in the day-to-day work and as a set of expectations regarding the level of quality to strive for.

Out of curiosity I decided to check out a number of publicly available definitions of done and count the percentage of items that clearly related to quality. Here are the results:

Standardization

A written, clearly-specified definition of done is essentially a standard. Standards provide many benefits when properly applied using a lean (pragmatic) mindset rather than a bureaucratic mindset. The proper mindset can be summarized using the following dichotomy: a standard represents the best way we know today of doing some activity within some context, yet each day we search for a better way so as to improve the standard. As implied by this summary, standards help form the foundation for future improvements. This is because following standards brings consistency which makes it easier to analyze the results of the work being done, determine the root cause of problems, and identify how to improve. For more on how standards help improvement efforts see my article on a framework for continuous improvement.

Standards have a number of other benefits. Checklists can be created from the standards for use by both the people doing the work (as a reminder of what to do), and by the people verifying the work (as a quality assurance check). Standards are helpful in training new team members. Having documented standards can also be helpful for achieving certain certifications like ISO or passing regulatory requirements such as Sarbanes-Oxley.

Most definitions of done do not mandate the specific step-by-step procedures to follow, but rather set expectations regarding the final product produced. To illustrate, a definition of done for coding features will likely mandate that certain kinds of testing be done (automated unit testing, functional testing, etc.), but will typically not mandate the sequence in which the testing and coding activities are done. (So unit tests could be written before the code as per test-driven development, or they could be written afterward - either approach will satisfy the definition of done.) Team may choose to standardize which procedures are used, but this is typically kept separate from a definition of done.

More Accurate Estimates

Having a definition of done helps estimation accuracy in two ways. First, it serves as a checklist of all the steps required to complete an activity that can likewise be used to estimate the effort remaining. When estimating the work left on a feature, for example, a definition of done can help ensure that time is estimated not just for the coding, but also for reviewing and testing.

Second, a definition of done makes it easier to track the actual progress, since activities reported as done are more likely (but not guaranteed, unfortunately) to actually be done. This means there will be less surprises in terms of additional work required for previously 'completed' assignments. These surprises, of course, significantly reduce estimation accuracy, so the more they can be eliminated the better the estimates.

Better Teamwork

The previous benefits apply when you are working on your own, but software development is more commonly a team activity. And in addition to these prior benefits, having a definition of done will make your team more effective. Many of the challenges of working in terms such as achieving unity of purpose, clear communication, and coordination of effort, benefit from having a definition of done. When team members are communicating about the work that has been and needs to be done, having a clear definition of done helps avoid misunderstandings. The definition functions, in essence, as a communications technique for ensuring that when team members use the word "done" they all mean the same thing.

For examples of what happens without a clear definition see this story of a team of five having three different viewpoints regarding done and this story of someone reporting a status of 'finished' that really wasn't.

Concluding Thoughts

This article has mostly discussed definition of done in the context of software development and specifically coding, but the concept is much more broadly applicable both within I.T. and beyond. Staying within I.T. I can see the benefits of having definitions of done for activities such as incident resolution by support staff, product high-level requirements definition, and pursuing sales. So I challenge you to look for opportunities to use a definition of done to help achieve superior performance.