There are a number of pragmatic practices that address this objective while avoiding the pitfalls afflicting formal traceability. I will start with a simpler form of traceability and then move on to practices seemingly unrelated to traceability.

Specification Highlighting to Track Test Coverage

A tester given a written specification to verify the software against must have some way of keeping track of their progress. One simple method is to highlight each statement in the specification as they test it. This is essentially tracking test coverage of the specification. I credit James Bach for this idea.

Testing Dashboard

A testing dashboard is great at illustrating overall testing progress and can be used as part of the summary in the final test report. To produce the dashboard first divided the system under test into test areas that usually correspond to features, components, or significant non-functional requirements. Then create a grid with the test areas as rows, and the columns report testing progress for each area. Key information to report per area is an assessment of quality (is this area ready to ship?), and the thoroughness of testing (test coverage and test effort). For an example and fuller explanation see James Bach's presentation on testing dashboards.

The testing dashboard is closest in form to a requirements traceability matrix - there's still a grid and something like requirements are listed down the grid. The big difference is that the dashboard shows a summary of testing for each area rather than listing individual tests and showing how they map.

Agile User Stories with Acceptance Tests

A common approach used by Agile methods is to divide requirements into fine-grained increments of business value called user stories and then define with examples or automated tests the criteria by which to accept that the story was correctly implemented. This takes a two-pronged approach to address traceability's objective. First, using fine-grained increments of functionality that are quickly developed minimizes the amount of work-in-progress that needs to be tracked and thus reduces the chance of mistakes. Second, the process produces tests for each user story early, sometimes before coding starts, so missing tests for a requirement is much less likely. Agile basically sufficiently changes how development is done to negate virtually all the sources of problems that would otherwise justify a requirements traceability matrix.

Task Tracking via Task Board

Popularized by Kanban but also used by many Agile teams, a task board is a practice for tracking the status of tasks. The board has multiple columns representing different status: the minimum set is usually "Not Started", "In Progress", and "Done". Tasks are placed on the board according to their status, and the team meets daily to discuss the tasks and update appropriately. I vastly prefer a physical task board on a wall, but for teams not co-located many online versions exist. Tasks are often quite fine-grained - one user story is often decomposed into multiple tasks.

Many teams include a status of "Testing" on the board, or use a symbol on the card to signify the completion of testing. Other quality control procedures such as code reviews can also be tracked. See the image below for an example of a task board with such states.

The combination of fine-grained tasks, daily updates, and tracking quality ensures that each requirement that is tackled by the team is properly developed. This achieves traceability's objective.

Conclusion

Despite formal traceability being too effort-intensive for too little value the underlying objective should not be ignored. I have described a variety of alternative practices that help ensure that requirements are properly developed and tested with a more modest level of effort.

Adopting Test Driven Development

The boldest idea I heard was in response to my question of how to encourage the adoption of test driven development (TDD). The answer consisted of this process:

1. Clearly communicate to the team the expectation that all production code will be produced with tests.
2. Wait and monitor code check-ins until you see code checked in without tests.
3. Delete this untested code and commit the change.
4. Communicate to the affected developer that you found some code that was not necessary because the tests still passed after removing it. Diabolical laughter optional :)

This sounds extreme, but the team I was talking to actually used this in the formative early stages of building their team and the use of TDD has become a standard practice on the team. I think applying this at the start of a project with a new team is the best timing and makes the most sense, but I may have to try it mid-project (my team, watch out :)

I also learned of a different approach to writing individual tests that basically involves starting at the end and working backwards. Start first with the asserts that specify the desired result, then write the execution of the method under test which covers how the result will be obtained, and then finally write the setup code which provides the necessary inputs. This approach is very much a pull approach (in the lean sense) - each step serves as the specification for what needs to be done in the next step. As such, it might be an easier approach for newcomers to unit testing.

Adopting Pair Programming

I asked about dealing with resistance to adopting pair programming, especially the tendency of pairs to collaborate jointly for only a short time before reverting to working separately, each at their own computer. The answer was to take away someones computer, which leaves them with no choice but to pair with someone else on the team.

Environment factors can also help or hinder pairing. The Calgary teams have their workstations set up to allow people to sit side-by-side in front of the same machine. I think this layout is very important for successful long-duration pairing, and I switched my own cubicle over to this layout a while ago. The Calgary teams, however, took this setup one step further and have a second keyboard and mouse for each of their workstations. This has several benefits in terms of encouraging pairing:

• It communicates visually the expectation that two people will work at this computer rather than one.
• It eliminates concerns over using another person's keyboard. The 'owner' of that workstation has their dedicated keyboard, with the second keyboard reserved for guests. Whether the concern is over germs or over intruding on someones property/space, having only one keyboard produces a reluctance on the part of the visitor to drive.
• A common problem in pairing is for the person who is not driving to become too passive. Providing a second keyboard gives them the opportunity to jump in at any point (even when the driver does not want to relinquish control), and having the keyboard waiting in front of them is a subtle reminder to remain actively involved.

Using Task Boards

All the teams use task boards consisting of cards in various columns on a whiteboard, wall, or even a window. My team has recently switched to using a task board and I really like it in terms of it visualizing the work in progress and serving as a hub for discussions, like in the daily scrum. Given my experience and its broad use in Calgary, I am going to more actively promote its use.

Every team seems to have variations in how the task boards are used, but there were some commonalities across the Calgary teams. All teams used large cards - about the size of half a normal 8.5 x 11 page. Compared to the small post-it-note-sized cards my team uses this is a lot more real estate. These cards are made from colored paper cut in half that allows different colors to be used for different types of activities (e.g. defects, stories, and tasks). One team printed out defect reports (from JIRA) on regular paper and folded them in half to attach them to the wall. Another team attached printed requirements (with tests) folded in half to user stories. The teams all had relatively few columns for their task board - usually not more than four consisting of "In Sprint", "In Progress", "In Testing", and "Done".

Professional Development

The Calgary teams have a very strong culture of professional development that manifests in a number of ways. The most visible manifestation is my seeing many recent software development or I.T. books scattered around the office, many of which are books on my reading list (I may have to do an office raid :).

What I consider the most significant manifestation of this culture is the existence of a professional development group for senior developers. This group holds a number of activities that include periodic presentations by members on technical topics, book reading groups to jointly go through particular books, and Friday noon viewings of videos relating to software development.

One of the technical leads had an interesting rationale for doing professional development: not doing it is essentially stealing from your clients in the future. If you spend five percent of your time doing professional development and improve by five percent per year, then you will break even after the first year and show a benefit in future years: your output will be higher than before, even after subtracting the five percent. So not doing professional development will leave your client shortchanged in the future. If you let that trend continue long enough, you will eventually lose your client.

Summary

I had a great time and enjoyed being challenged with new ideas and perspectives on software development. I believe that visiting other teams or companies and seeing first-hand how they do things is quite beneficial. I encourage you to look for opportunities to do this and I hope to do more of it in the future. Thanks CGI Calgary!

What is it about the spreadsheet that is so alluring? There is some value to studying it - I think of it as uploading the numbers into my subconscious to get an intuitive feel for where there might be obstacles or challenges that the team is facing now or will face shortly in the future. But the value of doing so quickly drops off and yet we keep staring. Part of the temptation I feel is to find ways to bump that daily or weekly burndown just a little higher by finding that task that is actually complete or is over-estimated. This is especially true if progress has not been good.

So what can we do to stop starting at the spreadsheet?

1. Remember that identifying obstacles is no good unless they are dealt with. Focus on removing or mitigating the obstacles instead.
2. Ask yourself what activity you can do that will best accelerate the team's progress. This might be helping a developer that is stuck on a technical issue, freeing up the time from a tester who has become a bottleneck, or clarifying requirements for upcoming work. More studying of the spreadsheet is never the answer. This is especially useful when the team's progress has been lower than expected.
3. Remember that minor errors in the spreadsheet are inconsequential in the grand scheme of things. Forgetting to burn down a task or two is dwarfed by the innate inaccuracies of the estimates you are using to track progress. If you miss updating a task this week, you will likely catch it next week.
4. Remind yourself that the spreadsheet is merely a tool, and that you are the master of the tool - not the other way around.
5. Throw away the spreadsheet and use a simpler approach - perhaps a paper burndown chart on the team wall. Just ensure you do not then start staring at the wall.

To those with a bureaucratic mindset process and audits are the answer to every problem: if something goes wrong, add more process and then audit to ensure it is followed.

Audits serve a purpose, but they do have drawbacks. Over-reliance on audits can actually cause negative consequences to the organization which are often not taken into account by those pushing for more auditing. Audits should be designed to minimize these drawbacks, and the organization should introduce additional mitigations as necessary. In fact, I would go as far as saying that the use of audits should be kept to a minimum.

So what are these drawbacks? I have organized them into the following categories which are explored in detail in the remainder of this article.

• Reduces Productivity
• Harms Organizational Culture
• Limits of Assessment

Reduces Productivity

I define productivity as the amount of value produced for stakeholders based on the effort expended. In other industries such as manufacturing with more concrete notions of value it is possible to effectively measure productivity. In I.T., however, this is difficult to do, and I have never heard of an I.T. audit that evaluates productivity, value produced, or even effort expended. Audits typically instead measure compliance to documented processes based on historical documentation regarding the work that was done. This disconnect inevitably leads to a reduction in productivity in the following ways:

Do work to pass audit rather than deliver value

Auditing documented evidence of following documented processes causes people to produce documentation, whether or not it provides value, in order to pass the audit. This is especially true when the work becomes about producing the documentation instead of producing value.

Assessing quality after the fact is wasteful

Two important concepts from lean thinking are that any time delay in a process is waste, and that quality should be built into the process to avoid the waste associated with undetected quality problems remaining in the work-in-progress. Performing an audit long after an activity has been performed violates both of these concepts. In some cases I have seen audits are performed months after the activity has been completed - for example, an audit of a software enhancement that has already been deployed to production. If problems are found, it is too late for that activity. And the follow-up to determine if the problem has been resolved typically does not happen until the next audit, which is an even longer delay.

Harms Organizational Culture

An organization in which formal audits are a common occurrence risks, in my opinion, serious harm to the organizational culture by shifting the culture towards a more bureaucratic, stagnant, and confrontational atmosphere. This can happen in a number of ways:

Shift of focus away from organizational objectives

People generally have difficulty focusing on more than one or two priorities at a time. When the focus is placed on passing audits, attention is diverted away from achieving the organization's objectives or purpose. This is similar to the problem faced by traditional project management - making scope, schedule, and budget the goal runs the risk of not actually meeting the business needs.

Compliance mindset rather than improvement mindset

Formal audits, especially when they are treated strictly by management, tends to lead to a fear-based compliance mindset. The unspoken message is "pass the audit, or else...". This is inimical to a mindset of continuous improvement, specifically the attitude of being willing to change the process when a better approach is found. In theory a process-centric mindset and a continuous improvement mindset can coexist - Toyota seems to be successful at this. But adding audits into the mix adds a rigidity or bureaucracy to processes that impedes continuous improvement. Worse is when people blindly follow process rather than question whether there is a better way.

Attitude of confrontation rather than collaboration

Formal audits almost always involve separate auditors external to the group being audited, and sometimes external to the organization. And these auditors are trained and motivated to find non-conformities - deviations from process. Auditors are not engaged to work with the group to understand what they have done and to help them do better. In fact, they are often instructed to remain aloof in order to remain impartial. This tends to lead to an attitude of confrontation rather than collaboration - an us-versus-them mentality - which means that any useful feedback that the auditors might have is at risk of being denied or ignored.

Discourages context-based expert judgement, creativity, and initiative

The Dreyfus model of skill acquisition explains the difference between beginners and experts. Having a clearly defined, step-by-step process to follow is essential for beginners, but has been demonstrated to actually reduce the performance of experts, who use their judgement based on the context at hand to determine what to do and how to do it.
Enforcing adherence to a common process also discourages creativity, initiative, and innovation. This can happen in a bureaucratic environment without audits, of course, but I believe that a regime of formal audits will exasperate the problem.

Limits of Assessment

Audits at their essence are about assessing a group with respect to some standard. This assessment has certain fundamental limitations due to the nature of audits, and over-reliance on audits risks developing an incomplete or inaccurate picture of reality. These limitations include:

Cannot assess intangibles

Audits only assess concrete items that are documented and completely ignore intangible aspects of work such as motivation, creativity, and initiative. (Some of these intangibles can be assessed by other 'formal' mechanisms - e.g. assessing motivation through surveys.) The corollary to the adage "you get what you measure" is that you risk not getting what you don't measure. This would not be a problem if the intangibles were not important. But software development is knowledge work, and the research is clear that people are by far the most significant factor determining productivity, far ahead of process. So relying on audits, which typically only look at process, ignores the more intangible, people factors.

Assessment is twice removed from objectives

Each group or organization has a purpose for existence, and objectives by which it tries to achieve that purpose. Audits do not assess this. Organizations put processes into place to support meeting these objectives. But audits, strictly speaking, do not assess this either. Audits assess the historical, documented evidence that the processes were followed. This means they are twice removed from the true objectives, which makes them an incredibly weak tool for helping an organization meet its objectives.

Audit results may be invalid

Audits suffer from four failure modes which can cause them to report invalid results:

1. The group is actually following the process being audited, but is not correctly producing the evidence.
2. The group deliberately deviates from the process in order to meet organizational objectives or in order to be more effective.
3. The group produces the documentation used as evidence for following the process without properly following the process. They will pass the audit, but there is a hidden problem. Rubber-stamping, where a review or approval is given without any actual check or thought, is an example of this.
4. The group properly follows the process and produces the evidence, but fails to achieve organizational objectives.

In conclusion, be very careful in how you use audits and watch out for the drawbacks. Maybe you should audit your use of audits :)

The ability to work independently and the amount of assistance provided to others both correlate negatively with task difficulty and correlate positively with the level of contribution to the team. This similarity of relationship allows us to combine both the independence and assistance measures into what I call "net team contribution". Similar in concept to a financial balance sheet, net team contribution is calculated as contribution assets - personal achievements and the help provided to others - minus contribution liabilities – the help that others had to provide.

I explained in my prior article that there are two thresholds of task difficulty: the level of difficulty at which you start to require assistance, and the level of difficulty at which the task is so excessively difficult for you that it is completely beyond your capabilities. Relating these thresholds to net team contribution results in four zones of contribution. The graph below provides a visual representation showing net team contribution as a function of task difficulty.

The four contribution zones are as follows:

1. Provide Assistance Zone: The task is easy for you to complete. You can easily see the way through obstacles faced by others and can effectively help them with their issues. Working primarily on such tasks yourself probably is not the best use of your time.
2. Independent Work Zone: the task is just easy enough that you can work on it independently with minimal assistance, but it is still somewhat challenging.
3. Help Required Zone: the task is difficult for you and you require some assistance to complete it. This zone starts at the first threshold of difficulty of requiring assistance.
4. Negative Contribution Zone: the task is very difficult for you. You require so much assistance from others on the task that they are essentially doing it for you. It is more cost-effective for others to simply complete the task themselves. This zone starts before reaching the second threshold of excessive difficulty.

Understanding these contribution zones and the underlying relationship between contribution and difficulty is helpful in ensuring a team is running smoothly. Is someone who has previously worked largely independently now struggling with a task? It is probably beyond their threshold of ease, so be sure to provide some assistance. If a developer is completely stuck and at a loss for how to proceed, the task may be completely beyond their abilities, in which case reassign it. When looking at the set of tasks to be tackled by the team within a single iteration, ensure there is a balance of complexity. Too few complex tasks leaves the more capable members under-utilized. Too many complex tasks is especially dangerous: it will result in the more capable members being too burdened by tasks near their level of ability to effectively provide the assistance required by the more junior members, while the juniors will be largely unable to make effective progress due to the complexity.

Achieving the full definition of done for a feature or release (often called being done-done) is surprisingly difficult. You may have heard the anecdotal rule of thumb that it takes 80% of the time to achieve 80% complete on a feature, and another 80% time-wise to achieve the remaining 20%. This is not due to using a strict definition of done. Anyone who has worked on an undisciplined waterfall project that has 'finished' coding the functionality and is now in test knows that sinking feeling of having defect after defect reported with no end in sight – and hence no predictability.

Using an imprecise or overly-simplistic definition of done will definitely impede predictability because you have only a vague idea of what the current state of your software is, and hence have high uncertainty regarding your progress to date.

Using a specific definition of done combined with the lean principle of minimizing work-in-progress greatly improves your knowledge of the current state of your software and hence your progress. Most of the uncertainty has been driven out by forcing work to done-done. This is not enough, however, to improve the predictability of your budget or schedule estimates for your outstanding work. You need to ensure your planned tasks and estimates include everything required to get to done-done.

On my last project, we tried just tracking coding tasks in the sprint plan. The remaining effort to get to done-done was treated as part of the general overhead that was used to calculate our velocity. This provided poor predictability in terms of both budget and schedule, especially as a feature neared code-complete. The estimate on the remaining coding tasks approached zero, but the time required for reviews, testing, and fixes remained roughly constant, thus exceeding what could be accounted for by velocity.

We then started to track most checklist items from our definition of done as individual tasks for each feature in addition to the regular coding tasks provided by the developer(s). This worked better, but we still suffered predictability problems as the feature went through review and test. Defects discovered had to be fixed, impacting the schedule and budget for the feature.

The next tweak we made was to allocate time up front for these fixes by adding an entry titled "Anticipated Issues" to each feature that provided an allocation of effort (budget, from which schedule was derived) that could be used to address defects, review corrections, poorly estimated tasks, and discovered work. The amount of effort allocated to this item was estimated per feature based on the size and complexity of the feature and the experience and quality level of the developers working on the feature.

Our most recent adjustment was to the front-end of feature development. Over several sprints, we noticed a trend that when developers started a new feature, especially one that was significantly new, they often had a day or even two where they did not make the expected progress on their tasks. We eventually determined that developers consistently had to spend time familiarizing themselves with the feature, especially learning the requirements and getting clarifications regarding how the feature was to be implemented, and that they virtually never allocated time in their tasks / estimates for this. So we added a new task to each feature named "Feature Clarification" to make this explicit.

By planning for all the components of our definition of done, we have achieved much better predictability. While our variance from our estimates per feature remained fairly high, the average variance across features was much closer to the original estimate, rather than being consistently over. As progress is made on a feature, we have much better estimates and better visibility into the outstanding work.

One solution to this problem that I have worked with and now recommend is to use a feature done checklist to track the completion of the definition of done for a specific feature. At its essence the checklist consists of the major elements of the definition of done listed as rows in a table with additional columns for the person verifying the completion of the item to enter their name and date. Here is a simple example:

You need to decide whether to use digital or printed checklists. I prefer using paper-based checklists for several reasons:

• I find it more meaningful to sign a piece of paper than to add my name electronically, so I tend to treat the completion of the checklist more seriously than I would an electronic form. This attitude may be due to my engineering background.
• A paper copy is more tangible and real. It can be waved around at a team meeting to celebrate the completion of a feature or posted on a team wall. It can be placed on a coworker’s keyboard in order to focus their attention on an outstanding item.

Evolving the Checklist

I have found it useful on occasion to evolve the basic feature done checklist beyond the team’s definition of done to provide support for the team’s development process. This is usually done as part of continuous improvement to resolve issues identified during retrospectives. Here are some examples:

• On one project we had issues with developers rushing into coding new features without fully understanding the requirements, so we added an initial checklist item for feature clarification to focus attention on the activity.
• For each item you can specify the role (e.g. developer, tester, architect) responsible for signing that it is done. I strongly recommend doing this for two reasons: it minimizes confusion over who signs off each item, and it minimizes the problem of having the developer of the feature quickly sign off all the checklist items without doing the necessary checking so they can say the feature is done.
• You can specify dependencies between the items. On one project we had a problem with people treating the items as a linear sequence of activities, so we defined dependencies to explicitly show that some activities could be done in parallel.
• You can add a feature start date to indicate when work began on the feature. This in combination with the feature done date allows you to calculate the duration of time spent on the feature. In lean terminology, this gives you the cycle time for the feature, which you want to minimize. When aggregated across the team, this defines the team’s overall throughput in completing features.

Checklist Challenges

I have encountered a number of challenges when using feature done checklists. Some of these issues are due to the actual checklists, but many are actually due to using a strict definition of done – the checklists just make the issue visible.

1. The checklist needs to be meaningful to the team - you must avoid the risk of it just becoming a form that people sign without doing the necessary work / checking that the item entails. (This is sometimes called rubber-stamping or pencil-whipping.) I have used many strategies to mitigate this risk:
   • Use a paper-based checklist rather than electronic.
   • Keep the checklist to a single page with as few items as possible for each role to complete.
   • For items involving ‘checking’ activities, assign these to roles / people whose primary responsibility is checking, separate from the developers doing the coding.
2. Extra effort is involved in getting a feature fully complete. This is often not accounted for in estimates, particularly if the estimate is provided by the developer. The time required for checking by other roles can be accommodated by having separate estimated tasks for these activities. What I find particularly challenging is accounting for extra time needed by the developer to resolve issues found in reviews or testing.
3. Having a through definition of done with multiple hand-offs of the feature to various roles will typically extend the duration of time required to complete the feature. The mitigation is to allow as many items as possible to be done concurrently. The best example is using pair programming as the means for doing the peer code review.
4. A single role / person with multiple review items per feature can end up becoming a bottleneck that ends up delaying feature completion. This is especially problematic if you are using a methodology like Scrum which focuses on getting features (user stories) done within the iteration – you tend to get a backlog of checklist items forming at the bottleneck near the end of the iteration which jeopardizes their completion.
5. One problem with using paper-based checklists is that they can get misplaced, or you end up looking for who has a form. This can be addressed by defining a location for checklists to be returned to. One option I like is to post them on a team wall, which provides the additional benefit of visibility.

Typical Definitions of Success

What are common definitions of success?

The fourth edition of the Project Management Body of Knowledge (PMBOK) frequently refers to project success, but does not appear to give a concrete definition which I find a little unsettling. All it seems to state on the subject is that the project charter should define the success criteria / objectives.

The Standish Group is well known for publishing their Chaos report that identifies leading causes of I.T. project failure. Success is defined as a software project that finishes on time, on budget, and within scope (e.g. with no missing functionality).

The article http://www.it-cortex.com/Stat_Failure_Rate.htm provides a number of different assessements of I.T. project success including the Standish group`s Chaos report that generally involve time, budget, and scope.

Formal definitions of success may seem too theoretical. Perhaps more relevant is how are projects measured and assessed within the organization, both in terms of the project status while it is in progress, and the evaluation of project success after the conclusion of the project. While I cannot speak for your organization, the ones I have worked for, observed, or heard case studies about all measure time and budget, and usually measure scope.

Many definitions of software project success that I was able to find generally correspond to the following definition: "A high-quality product that satisfies agreed requirements within time and budget" taken from the book A Practical Approach to Software Quality by Gerard O'Reagan on page 215. For the remainder of this article I refer to this as the typical definition of success.

Success and Failure Scenarios

The typical definition of success may be quite popular and common, but is it useful, accurate, or comprehensive? To put it to the test I present a number of project scenarios – some real, some hypothetical – and see whether the typical definition of success matches what the situation suggests.

Death March

The project schedule and/or scope was overly aggressive. The project team worked long hours for an extended period of time in order to get the software completed within budget and schedule. A few team members quit or burnt out during the project, and most resigned shortly after the project finished.

Strictly speaking this is a hypothetical scenario, but I have heard of a few projects at various game companies that have come close. The book Death March by Edward Yourdon would not have become a classic if this scenario was not a common occurance within the industry.

Analysis: While the typical definition would call this project a success, I consider it a failure, and I would hope most reasonable organizations would as well. Definitions of success need to consider the project's impact on the team.

Unsatisfied Users

The project completed within schedule and budget with slightly more functionality than the original scope. But the users felt that their requirements had been misunderstood by the project team and that the software did not meet their needs.

This scenario is based on a project I was involved with a few years ago.

Analysis: My initial reaction at hearing of unsatisfied users is to label the project a failure, even though it qualifies as a success according to the typical definition. Definitions of success should consider the satisfaction of the customer and end users of the software.

However, given my experience with this scenario, I know that an important question to ask is whether the users' dissatisfaction was reasonable- i.e. did they know what they wanted, and was it achievable? I am reminded of a Dilbert cartoon where Dilbert is gathering requiremnts from a marketing person who starts fantasizing about a telepathic user interface. Some people you cannot make happy, no matter how hard you try.

Marketplace Failure

Company executives considered the project a success because it had been run by the book and the final system was very similar to the initial specification. But an external expert who evaluated the project found that it scored poorly in quality and productivity and faired poorly in the marketplace. Ironically, a second project that company executives considered a failure because the project was continually changing in efforts to respond to changes in the marketplace ended up quite successful in the marketplace.

This scenario is described in page 30 of the book Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck and comes from a paper by Alan MacCormack, a professor at Harvard who was the outside expert.

Analysis: There are several lessons here. The first is that definitions of success should consider how the product performs in the marketplace. This relates to the prior point regarding satisfaction of customers / end users - the more satisfied the consumer is, the more likely the product will excell in the marketplace. One question to consider, however, is whether the success of the project should be completely dependent on the success of the software that is produced. One can easily imagine scenarios where a team does a great job producing a great product, but it fails in the marketplace for reasons outside their control (e.g. bad marketing).

The second lesson is to observe the danger of having a wrong definition of success. Those company executives, who presumably make decisions to approve and fund projects and reward project teams, evaluated project success completely opposite to the level of success in the marketplace. The risk is that these executives cancel the 'bad' projects and fund more projects like the 'good' one, and put their company out of business as a result.

Late but Leader

The project was years late before it was finally delivered: at no point in time was the project estimated to take more than about one year, but it actually took just over five years to ship. The software, however, became number one in the marketplace and a flagship product for the company.

This story is told in the book Rapid Development by Steve McConnell (pages 207 - 208) about Microsoft Word for Windows version 1.0. I assume you have heard of this piece of software :)

Analysis: This scenario is essentially the opposite of the prior scenario and thus raises the same question about whether product success means project success. In this case, given how badly the project was estimated and the negative impact that the overly-aggressive schedule had on the team's efforts, I would be hard-pressed to call the project itself a success, although the product itself was extremely successful.

Stopped for Higher Priority

The project was stopped by management part-way through with only partial functionality delivered in order for the team to start a new, much higher priority project to take advantage of an excellent opportunity in the marketplace.

This scenario is hypothetical.

Analysis: The typical definition of success would call this project a failure. But why, exactly? Because not all the functionality was delivered? If management determined that continuing to work on the remaining functionality was less valuable than starting the new project, then wouldn't the really failure have been to continue with the project (assuming no other people were available to start the new project)? When a project is started, certain decisions are made regarding its scope, schedule, and budget, usually based on limited information. If these decisions are changed later on the basis of more accurate information, how is this a failure? Another point I've hinted at here is that worthwhile projects must be delivering value to some customer or stakeholder, and defintiions of success should consider this.

Cancelled

The project was cancelled after a few months after it became clear that the new leading-edge technology being piloted was not working out and that the project's expected return on investment would not be achieved.

This scenario is hypothetical.

Analysis: This continues the theme of the last scenario regarding the delivery of value. Projects that are not delivering value should be stopped (or fixed). But is a cancelled project a failure? You might be tempted to say yes. I was. But consider that the typical definition of success considers cancelled projects failures, whereas if such a project was left running to its conclusion it would be considered a success, irrespective of the value produced. People try to avoid failure, so calling this scenario a failure motivates people to avoid it and continue down the path of not providing value. But is it right to call this project a success? One could argue that it did provide value in terms of the organization learning that the new technology was unworkable. If evaluating the technology was one of the primary goals of the project, then that goal was definitely achieved.

Simultaneous Success and Failure

The project was considered a failure by management due to not meeting its estimates (it was 400% over budget), but a majority of the developers considered it their most successful project.

This scenario is told in the book Facts and Fallacies of Software Engineering by Robert Glass, page 39, and comes from research performed by K.R. Linberg.

Analysis: Robert Glass's analysis of this situation identifies that one of the underlying factors behind this disagreement is an "unspoken controversy ... about what constitutes project success." (page 41). My view is that different stakeholders can hold widely differing views of success. (And yes, I consider the project team to be a significant stakeholder in the project - they are the ones dedicating a portion of their lives to it.) Definitions of success should take into consideration these different views, but how do you reconcile them when they conflict? Is one stakeholder's view better or more correct than anothers? This goes back to my prior discussion on customer and user satisfaction, and what happens if they are unreasonable. The same can be said of management's expectations - if the initial schedule or budget is completely unrealistic (which the developers in this scenario did identify as a reason the project was late), then why shouldn't the management view of success be disregarded?

No Budget or No Schedule

Some projects, for all intents and purposes, have no budget or no schedule. The best example is open source projects. Some of bigger open source projects do have a target date for their next release, but keep the actual release date flexible based on when the software is ready. And virtually all open source projects have no budget at all.

Analysis: The typical definition of success is unable to even evaluate the success or failure of such projects, which seems wrong for a supposedly universal definition. You might try to claim that open source development does not consitute a project, but any given release of open source software has a clearly defined start and end, with activities being performed within that duration to achieve the goal of shipping that release. That sounds like a project to me. How would you judge the success of open source development if there is no budget, a fluid schedule (maybe), and maybe not even a defined scope? I think the answer lies in determining whether the stakeholders are satisfied and whether value is being produced for these stakeholders.

Conclusion

I purposely selected scenarios that challenge the typical definition of sucesss in order to highlight its flaws and to identify considerations that can lead towards a better definition of success. The essential underlying problem, I believe, is a broken paradigm, which I illustrate by rewording the typical definition of success in story form:

When we are about to start a project is when we know the least about it, which means that our estimates will be the worst possible. Let us guess, up front, precisely the scope (functionality) we want to deliver and then guess the budget and schedule required to deliver that scope. These guesses may then be arbitrarily revised downward for budget and schedule and upward for scope for political reasons. Let us then commit to these guesses. If we do not achieve them our project will be judged a failure, no matter what we might learn throughout the project.

Hopefully worded in that fashion it is clear that there is something fundamentally disfunctional about evaluating projects in this manner. My next article in this series on success will focus on coming up with a better definition of success.

P.S. I must point out that I am being somewhat unfair in using the term "typical definition of success" as others have had similar thoughts to myself and have come forward with alternate definitions. See for example Scott Ambler's analysis.

Defining what makes for a successful project / product is critically important at every level: the individual, the team, the organization, and even the entire I.T. industry. (In the remainder of this article I will refer solely to projects with the understanding that the material applies as well to ongoing work on software products.)

Individual

Individual performance is often evaluated at least in part by the success or failure of the projects you work on, more so if you are in a leadership or management role. This can impact your compensation (raises, bonuses), future assignments and promotions, and even your job (if the project is a bad failure).

Being clear about the project’s definition of success should help guide you in the day-to-day decisions you make and actions you take on the project. This provides two benefits: you are more likely to be personally judged a success (even if the project fails), and the project is more likely to be successful.

Team

The choices to make in assembling a team – how many people will be on it, and what skill levels & abilities will they have – are significantly influenced by the project’s definition of success. For example, if minimal budget expenditure is a critical success factor then a small team should be used.

Once the team is assembled having a clear definition of success shared by the team members helps provide coordination of activity rather than disjointed efforts.

The practices & methods used by the team should be selected and tailored to maximize the likelihood and magnitude of success.

Organization

Within an organization that produces software decisions must be made regarding what projects to proceed or continue with and what level of funding and people to allocate to these projects. In a rational organization these decisions should be based on a cost-benefit analysis which should, in turn, be linked to the project’s definition of success. For example, if a project is conceived of to produce software targeted to hit the market in time for the Christmas selling season then hitting the delivery date becomes a significant factor of project success.

I.T. Industry

Our favorite pastime within the I.T. industry may well be promoting or debating the merits of various programming languages, environments, tools, practices, and methods. But what is the basis for making these evaluations? Presumably this is based on the likelihood of the tool, practice, or method making the project more successful (or less likely to fail). This evaluation should depend, therefore, on the definition of success being used, but this is seldom defined explicitly as part of the promotion / evaluation being done. (I am as guilty of this as everyone else.)

One notable exception is the book Rapid Development by Steve McConnell. Part III of the book lists 27 ‘best’ practices. For each practice a section on efficacy is provided that evaluates the practice based on three separate factors: potential reduction from nominal schedule, improvement in progress visibility, and effect on schedule risk. (Other project success factors such as quality or cost were omitted because the book focuses on minimizing schedule – thus the name Rapid Development.) McConnell explicitly advocates “selecting rapid-development practices that meet the needs of your specific project” (page 396).

Some might argue that virtually all projects share the same definition of success (usually along the lines of delivering required functionality with sufficient quality on time and on budget), and this then allows for meaningful discussion of 'best' practices independent of the definition of project success. I disagree with this position, but will save further discussion for a subsequent article: Problems with Typical Definitions of Project Success

Dealing with Uncertainty

Mike characterized traditional approaches as trying to manage out uncertainty in the effort to achieve predictable results. This strategy essentially involves determining up front the key aspects of the project (scope, budget, and schedule) and then seeking to minimize changes to it throughout the project. Unfortunately, virtually all software development is faced with a multitude of uncertainties. Trying to drive them out means conflicting with reality, which leads to a common litany of project problems such as: over budget, over schedule, scope achieved but business not satisfied (business objectives not met), and excessive cost for realized value.

Agile adopts the opposite philosophy: manage for uncertainty. Kent Beck used the term "embrace change" when first introducing Extreme Programming, and this is a common goal of all Agile methods. This is achieved through several strategies:

• Requirements are defined and implemented throughout the project, making it trivial to accommodate changes. The next section below discusses this in more detail.
• Providing frequent feedback via working software mitigates many of the risks of traditional development projects.
• Prioritizing user stories and first implementing the ones with the highest value / highest risk maximizes the return on investment and mitigates a number of common risks.
• Often Agile projects explicitly allow for the project to be stopped or brought to a close at any point if the business owner determines that there is no benefit in proceeding with the remaining requirements.

One of Mike's though-provoking points was that using an Agile method is a risk-mitigation strategy that mitigates many of the common risks associated with software development projects.

Deriving instead of Fixing Scope

One key aspect of project management is managing the iron triangle of scope, cost (budget), and time (schedule). In a traditional waterfall project the scope is defined first, the effort required to implement this scope is then estimated, and finally the budget and time required are derived from this estimate. This approach is maintained throughout the project: controls are often put in place to minimize changes to the scope and changes that are deemed necessary then typically require adjustments to the budget and/or schedule.

Fixing the scope up front is inherently flawed due to the nature of software development:

• Business requirements almost always change over time. I have seen statistics mentioned on the order of 1% change in requirements per month.
• Software development is inherently a learning activity. The business typically gains clarity about their requirements as they are analyzed and implemented. Insight is also gained into how the solution / system can best meet these requirements.
• One technique from lean software development is set-based concurrent engineering, which aims to defer decisions to the last possible moment when the most information is available. This helps ensure that the best possible decision is made and hence minimizes waste caused by bad decisions (one of the main objectives of lean). Fixing scope up front requires making decisions at the worst possible time – when the least is known.

Agile addresses this problem by making scope flexible. While an Agile project can (and usually does) start with a high-level backlog of features or epics, the specific user stories selected for implementation are determined throughout the project, iteration by iteration. I have heard some say that Agile makes the project scope easy to change, but in reality the scope is not being changed at all – it is being defined throughout the project. The total scope achieved in a project is hence a function of the project schedule: the more iterations and releases available, the more that can be done. Going back to the triangle of scope, cost, and time, this means that time is the main variable being controlled, from which budget and scope can be derived.

Conclusion

I really enjoyed the clarity of thinking Mike brought to the topic of Agile project management. Considering that Mike has given this presentation at major software development conferences that require $$$ to attend, I really appreciated having the opportunity to see it for free in my home town. I am a little surprised that there are not more people taking advantage of these opportunities. If you live in the Edmonton area I strongly encourage you to attend the Agile Edmonton meetings.