Shakespeare quotes aside, every software development project has to make decisions on how many defects to fix and which ones to leave alone prior to shipping. While I have seldom seen this question debated within projects, the advice from industry thought leaders varies considerably. The Agile and Lean methods of software development in particular have somewhat opposing perspectives.

I believe that considering both sides of this question provides a fuller understanding of the issues and better equips us to answer appropriately. Therefore in the two sections below I explore the reasons behind both sides of the debate.

To Fix

• Shipping poor quality, defect-ridden code can upset users, turn away customers, and lead to a hard-to-shake bad reputation.
• The decision that a feature is worth developing is made with the expectation that it will work correctly. So any defects found in a feature means that the feature is still incomplete until these issues are fixed.
• Defects provide feedback regarding the development process. Each defect represents an opportunity to do a root cause analysis of what led to the defect and put countermeasures in place to prevent re-occurrence. The Lean mindset of "Stop the line" demands that new development be put on hold to fix newly discovered defects.
• Defects introduce the risk of compounding quality problems. The impact of a defect can be more significant than initially realized. Defects can be inadvertently replicated in other parts of the system. Enhancing components with too many defects can slow progress to a halt, as the system becomes essentially a shifting quicksand that is too unstable to work on. Constantly fixing defects helps maintain a high velocity of development over time.
• To mitigate risks in not fixing defects, each defect needs to be analyzed to understand its impact, cause, and required changes to fix. But after performing this analysis most of the work is usually done - the fix is relatively straightforward. Waiting to decide later to fix the defect (e.g. in a subsequent release) causes all the knowledge gained in the analysis to decay over time which is wasteful (in the Lean sense).

Not To Fix

• Significantly delaying the release of software to fix all defects leads to a loss of immediate revenue and potentially loss of market share due to competitors beating you to market. So you cannot afford to wait to fix all defects.
• The entrepreneurial mindset, especially for startups, is to ship early to get feedback from paying customers. Perfection is the enemy of the good.
• Under at least some versions of Scrum, defects are considered new tasks that are added to the product backlog to be prioritized by the product owner. This prioritization is based on the defect's impact (severity and likelihood of occurrence) and the effort required to fix it. Many minor defects will therefore likely never be fixed as new functionality will typically be of higher value.
• Stopping to analyze and fix defects disrupts developers who are in the middle of working on other functionality and is wasteful.
• Fixing defects in functionality that is already otherwise finished development and testing will require additional regression testing. Not fixing now and waiting until enhancements to this functionality are needed minimizes the extra effort required.

Conclusion

Shakespeare was wrong. There is actually a third perspective regarding whether or not to fix defects: avoid the question as much as possible by focusing on defect prevention. The Lean mindset of building quality in avoids all the waste associated with finding, analyzing, and fixing defects and should be our preferred approach. Only when it fails and the occasional defect is introduced do we then have to answer the question.

Role-Based Attitudes

I will first answer this question on a role by role basis, starting with the role of tester. Since one of the primary objectives and professional skills of testers is to find defects, I would expect them to generally be happy when they find problems. The more significant (e.g. critical) the defect or more tricky to find, the happier I would expect them to be. If the defects are trivially easy to find, or are so serious that they prevent further testing, then I could see testers getting upset because they are prevented from exercising their craft to the fullest level of their capabilities.

What about developers? When developers find defects in other people's code, perhaps via a code review, this is fairly similar to the tester role above so I would expect developers should generally be happy in this scenario.

When a developer finds a defect in their own code as part of the development process - e.g. by having an automated test that unexpected fails after a code change then how should they feel? It is only natural to feel some disappointment at having made a mistake, but I do not believe this is the best reaction. In fact, it may be harmful. If you experience mild distress or pain whenever you find your own defects, you may subconsciously start to try to avoid the pain by, for example, doing less testing. So I believe a better reaction is to be happy when you find your own defects. This may seem challenging to do, but consider these rationale:

• Finding the defect validates your skill at performing whatever defect detection activity you were doing - whether it was automated testing or personal code review - and reinforces the value of performing that activity.
• Discovering this defect provides a learning opportunity on how you might better prevent or detect this type of defect in the future. Learning opportunities are good.
• As a professional, you want to pass along high quality code. Every defect you find in your own code is one less defect others will find, thus raising the quality of your final output.

The final scenario to consider is when someone else finds defects in the developer's code. A typical example is when the developer considers a feature finished and a tester is testing it. Now this must be bad, right? Even more than the prior scenario it seems natural to feel disappointment. Each defect that is discovered is essentially downgrading the quality of what you have produced. However, some of the reasons we listed previously for feeling positive still apply. In particular if the defect was still found within the team rather than being passed along to the end user or client, then this is a good thing. It can be hard to see this as a developer when it is your code at fault.

So let us consider the final role: that of the team lead or manager who is supervising activity rather than specifically doing coding or testing. This role has no personal stake in individual defects, but instead is typically concerned about the broader implications to the project or product. Assuming that high quality is a key objective, how should such individuals feel about defects? One tendency is to react negatively because of the extra effort and schedule time required to fix defects. Finding too many defects can also lead to alarm bells over the level of quality. I feel that since the team lead or manager essentially represents the whole team, their reaction should correspond to the perspective of the entire team.

Team-Based Attitudes

So what should the attitude of the team as a whole be to discovering defects? To properly answer this question we need to know what the team's objective is. I am going to assume it is to build working software that is being used and meeting users' needs. The existence of defects is clearly bad since they threaten this objective. But what about the discovery of defects? Discovering a defect means the team gains more information about the state of the software which it will typically use to improve the software by fixing the defect, and which the team can also use as feedback to improve.

Discovered defects exist prior to being found, but the team does not know about them until the time of discovery. This means that the team should experience simultaneous conflicting reactions: happy that the defect was discovered, yet unhappy that the defect exists.

Individual Attitudes

So what should the attitude of the various individuals on a team be towards defects? Should it be based on their role? I believe that in an ideal team every individual will understand and adopt the common objectives of the team. This means that everyone's attitude towards defects should ideally be the same, no matter what the role.

As I have observed, in practice attitudes tend to be quite far from this ideal state. People maintain their own personal objectives and viewpoint in addition to or instead of the team's. Often they do not appreciate the distinction between discovering defects and the existence of defects. To correct this I believe team members need to be clear on the team's objectives and understand how the existence and discovery of defects impacts these objectives. I hope this article will help in that regard.

• Measuring Defect Potentials and Defect Removal (pdf)
• Software Assessments, Benchmarks, and Best Practices
• Estimating Software Costs: Bringing Realism to Estimating

The information I provide in the remainder of this article comes primarily from these references.

Estimating Software Size

The first step in Caper's approach is to estimate the size of the software. His preferred size metric is function points, which are a language/technology neutral evaluation of the business functionality based primarily on an assessment of program inputs, program outputs, and data storage. Another common metric is logical lines of code, often abbreviated as SLOC. KLOC represents 1000 lines of code.

Lines of code are a convenient metric in that they can be measured automatically with a tool, whereas function points required a trained function point counter. Nevertheless, function points are superiour in several key ways:

• Roughly fifty percent of defects are due to problems with requirements or design rather than coding, for which metrics in terms of lines of code do not make much sense. In particular, different implementations of the same functionality can have significant variances in the lines of code required by up to a factor of four. A longer implementation will have fewer requirement or design defects per KLOC and thus artificially seem to have higher quality, when in reality the code is simply bloated. (This is related to the issue of measuring developer productivity by lines of code produced.)
• The use of multiple languages complicates source code counts, and this is far more common than one may expect. Even a simple web application typically includes JavaScript, HTML, CSS, and perhaps SQL in addition to the primary language (e.g. Java). Function points are language-independent.
• Function points can be determined once the design is known - lines of code require waiting till coding be completed. This allows function points to be useful for planning and estimation purposes much earlier.

Since Caper prefers function points, the defect metrics he provides in his writings are typically expressed using this measure. While I understood the reasons, I found these metrics difficult to apply because I had no idea of what the function point counts were of the software I was working on. So I was pleased to discover that benchmarks such as this one exist to convert between function points and lines of code for various languages. One function point on average is roughly 50 logical lines of Java code. I use this conversion factor in the sections below.

Defect Potential

All software has the potential of having defects. Defect potential is a measurement of the expected number of defects in a particular piece of software. This is also called the injection rate - the number of defects being introduced throughout development. The primary factor determining the number of defects is the size of the application in function points. The maturity - experience, skill, and attention to quality - of the development team is another key factor in determining defect potential. The following table specifies how to calculate the expected number of defects given these two factors.

Some key defect prevention activities contributing to reduced defect potential are:

• Close customer collaboration during requirements / design (e.g. JAD sessions)
• Prototyping
• Feedback / learning from design and code reviews

The above metrics assume a linear relationship between defects and size, but as a system gets larger there are typicaly more interactions between pieces and more complexity, and thus a greater likelihood of defects than a linear increase would suggest. For very large systems, a more accurate metric is as follows: number of defects = function points raised to an exponent. For average organizations, use 1.25 as the exponent. (Good organizations can lower this to 1.15, while poor-performing organizations have this elevated to 1.35.)

Defects can be categorized by origin - the type of activity that produced the defect. The table below shows this breakdown for average organizations.

Defect Removal

Defect removal is the identification and elimination of defects after they are introduced. The cumulative defect removal rate or defect removal efficiency of a development project is calculated as the number of defects eliminated prior to the release to production divided by the total number of defects found after 90 days of production use.

The following table shows how the defect removal rate varies with the maturity level of the team, just like defect potential, and shows the expected number of post-release defects based on the defect potential and removal metrics.

Removal efficiency varies for defects of different origins, as the following table shows using statistics for average organizations.

Quality control procedures such as testing and reviews (inspections) vary in their effectiveness at removing defects as illustrated in the following table.

Peak defect removal rates for a given activity are typically obtained only through the use of skilled, experienced staff who take a rigourous, disciplined approach to performing the activity. As an example consider unit testing. As it is normally performed it has a 25% defect removal rate. But an experienced developer following test-driven development will achieve nearly 100% code coverage and typically write better tests, leading to a higher 50% defect removal rate.

Predicting Defect Levels

The overall or cumulative defect removal rate for a develoment effort can be calculated by aggregating together the individual defect removal rates of the quality control procedures used by the team as follows: Cumulative removal rate = 1 - the product across all procedures of (1 - individual removal rate per procedure).

For example, if a team uses only unit testing (25% removal), functional testing (30%), and regression testing (20%), then the cumulative rate = 1 - (1-0.25) * (1-0.30) * (1-0.20) = 0.58 or 58%.

The overall number of defects in production (or UAT) can be calculated using the defect potential of the team to determine the expected number of defects introduced, and using the cumulative defect removal rate to determine the number of defects remaining. Approximately 25% of defects will be high severity.

For example, an average organization injecting 1 defect per 10 lines of code for a 25 KLOC application will end up introducing a total of 2500 defects. Given a cumulative defect removal rate of 95%, this means that 2375 defects will be found, leaving 125 defects remaining of which 31 can be expected to be high severity.

You must be careful, however, that the cost of these defect prevention practices does not become excessive. That would introduce a different type of waste – non-value adding process. The waterfall method of software development is an example of this. One of the principles behind waterfall is that careful requirements analysis and design will minimize downstream defects during coding and testing. Put another way, it is a good idea to understand what you need to build before you start building it. The problems with waterfall arise from going to extremes in applying this principle. Requirements analysis is done up front for the entire project as a big batch based on the theory that it minimizes rework due to future change, but in reality the constant pace of requirement changes plus the learning that occurs throughout the project will result in increasing amounts of change the longer the time spent doing requirements. In contrast, Scrum and Kanban apply this principle using a balanced approach – project level requirements are done at a high level, and the more detailed analysis is done on individual user stories just prior to implementing them. (See for example the article Scrum and CMMI – Going from Good to Great: are you ready-ready to be done-done?.)

In order to effectively adopt a defect prevention practice two pieces of information are needed:

1. Specific, actionable steps to apply the practice.
2. The expected benefit. What categories of defects does the practice intend to prevent? This helps determine when to apply the practice and helps to evaluate it after adoption to assess its effectiveness.

If we consider the idea of careful requirements analysis and design mentioned above as a prevention practice, the benefits are fairly clear - prevent requirement and design based errors - but specific actionable steps are missing so it does not qualify. (In fact, this is one of the contributing factors why waterfall projects can end up in the analysis paralysis anti-pattern.)

Now that the groundwork has been laid I can present some specific defect prevention practices. This is not a comprehensive list – many other practices are possible. The practices I have chosen to discuss are ones that I have used and am confident that they work.

Use Understood Methods Rule

The basic formulation of the rule is quite simple: when coding a method only invoke other methods whose behavior you clearly understand and are confident will work the way you want. I have written a separate article providing specific guidance on how to apply this rule.

This practice generally aims to prevent interface errors - which I define generally as defects between two separate pieces of code. Research suggests that a significant proportion of defects are due to these kinds of errors (See for example the paper An Empirical Study of Software Interface Faults.)

I find this rule particularly valuable when applied to the invocation of methods between classes and especially between components. In this case it helps prevent integration errors which are usually not caught by unit testing.

Design by Contract

The idea of design by contract is to precisely specify the behavior of methods to help ensure that they are invoked correctly by callers and that the callers receive the results they are expecting. This is done by precisely specifying the preconditions and postconditions of methods. The chief proponent of design by contract is Bertrand Meyer, whose book Object-Oriented Software Construction is a classic on the topic.

Method preconditions are conditions that must be true in order for the method to successfully execute and fulfill its postconditions. Preconditions are most commonly applied to method arguments. For example, a method to convert a string to a date might have the precondition that the string argument must not be null. Preconditions can also be applied to the state within the class or even external state. For example, a method to delete a particular object from the database might have the precondition that the object exists within the database.

Method postconditions are conditions that the method guarantees to be true after execution, assuming the preconditions are met. Postconditions are most commonly applied to the return value of methods, but like preconditions can also be applied to the state within the class or external state. Returning to the example of a method that converts a string to a date, such a method could have two postconditions. First, that the method will return a corresponding date object that is not null if the input is a string in a valid date format, and second that the method will throw a specified exception if the input does not correspond to a valid date format.

The combination of preconditions and postconditions forms in essence a contract between the method and its caller. The caller promises to fulfill the preconditions in exchange for the method guaranteeing that the postconditions will be met.

Despite the fact that the name of this practice contains the word "design", this approach does not require a separate up-front design of each method. The goal is to have a clear specification of behavior once the method is finished – how you arrive at it is not important to this practice. I tend to start with an initial idea for a method’s contract that I evolve as I write tests and implement the method’s logic using test-driven development.

There are several options for specifying pre- and post- conditions. Some teams rely solely on their automated unit tests to serve as the specification, but I prefer a more concise specification provided as part of the method definition. In Java I typically use JavaDoc to document pre- and post- conditions and programmatically check argument preconditions at the start of the method. I typically formally specify pre- and post- conditions only on methods that are intended for use by other classes or components. In Java, this is typically public and protected methods of interfaces and classes.

This practice is very closely related to the Use Understood Methods Rule, and they go hand-in-hand. Knowing a method’s pre- and post- conditions is necessary to fully understanding it. As I stated above, I tend to only apply formal design by contract to methods intended for use outside the class in question, which means this practice is really aimed at preventing integration defects.

Defensive Coding

Defensive coding is named after the practice of defensive driving and is based on the same mindset of expecting problems to occur and actively taking precautions to avoid them. Defensive coding is applied by adopting a language-specific set of idioms that minimize or prevent common coding errors when using the language. These idioms are often reflected in coding standards.

Here are some examples of defensive coding idioms for Java:

• When comparing if a variable is equal to a constant, put the constant first. This avoids a potential null-pointer exception (if the variable is null) by invoking the equals() method on the constant, which is never null.

  public boolean isAdmin(String userId) {
    String constant = "admin";
    return constant.equals(userId); // Instead of userId.equals(constant)
  }
  

• Always use braces to define a block of code for an if, else, while, for, or do statement, even if the block contains only a single line of code. This avoids the problem of later adding a second line of code indented to the same level as the first and mistakenly thinking it will invoked as part of the block.

  public void addOptions(String userId) {
    if (isAdmin(userId)) {
      addAdminOptions();
    } else {
      addRegularUserOptions();
    }
  }
  

• Use the Java 5 for-each construct rather than using a loop index variable to manually iterate through a list. This avoids the problem of having an off-by-one error in constructing the loop.

  public void processOptions(List options) {
    for (Option option : options) {
      option.process();
    }
  }
  

Defensive coding aims to minimize coding errors, both at the time of coding and in the future when the code is being modified by others. While these types of errors are typically easily detected by unit testing, I find that using these idioms (after the initial adoption) takes virtually no effort or thought on my part, making them literally a no-brainer to use.

Example-Based Requirements

The idea behind this practice is to express requirements as much as possible in terms of concrete examples rather than the generalized wording typically used in use cases and lists of business rules which is almost always ambiguous. I have written a separate article providing further details on example-based requirements which includes a specific example. :)

The practice of example-based requirements aims at minimizing requirement errors, particularly errors due to misunderstanding or misinterpreting. The examples should also be used as acceptance test cases, in which case they help detect design or coding errors (although unit tests should identify most of these first).

Conclusion

I encourage you to choose one or more of these practices to adopt in your current development work. There will be extra effort initially to understand and become comfortable with a given practice, but this will decline over time as you achieve mastery of it.

Too many developers think their job is complete once a feature has been coded. Sometimes they think that it is the tester’s job to find defects. Sometimes they think defects in released code are unavoidable and normal, so not worth worrying about. Sometimes they believe their code is perfect - it cannot possibly have defects. I encounter developers with these attitudes with unfortunate frequency. I also encounter development managers who are surprised to encounter such attitudes. A while back I talked to one manager who was shocked to learn than one group of developers under her were assuming their code worked if it compiled successfully - there were no reviews or any sort of testing being done. So I hope with this article to raise the awareness amongst developers that coding is simply not enough to produce working software, and to raise the awareness amongst development managers that they need to ensure the appropriate systems are in place to support this.

The reality is that even the most diligent developers inject defects into their code at a surprisingly high rate. Defect rates are often defined as the ratio of the number of defects per one thousand lines of code (KLOC). Industry statistics on defect rates are rather hard to find and vary significantly, partly because the definition of defect used varies. Several studies have reported defect rates in the range of 10 to 100 defects per KLOC as reported in the book Best Kept Secrets of Peer Code Review. This works out to one defect per 10 to 100 lines of code.

On my most recent project I decided to calculate the defect rate for a particularly error-prone feature. Counting only defects found by independent testers after code reviews and unit testing were done, and using a KLOC count not including comments or blank lines, this feature had 20 defects for roughly 850 lines of code which is a defect rate of 24 defects per KLOC, or one defect for every 40 lines of code. This may seem reasonable, but remember that this is after multiple code reviews and automated unit testing have already found and eliminated a number of defects. (How many I do not know as these kinds of defects are not tracked.) And there still may be yet-to-be-found defects still lurking in this code. So the actual defect injection ratio is higher, perhaps much higher.

Defect rates have such a wide variance, even between developers working on the same code base, that it is unfortunately not a reliable metric for predicting defect counts. My main point in discussing them is to emphasize just how frequently defects are introduced.

Coding, therefore, is simply not enough. Every developer needs to have a personal system for preventing and eliminating defects, which should integrate into the system / processes used by the development team to produce high-quality working software. For ideas on how to assemble such a system check out my definition of done that identifies a number of defect elimination activities.

FindBugs is a Java static analysis tool that scans compiled java code for potential defects and bad programming practices. Think of it as the Java compiler on steroids: it operates in roughly the same fashion but reports on a much larger set of errors and warnings. Static analysis makes a great complement to automated unit tests. Unit tests require effort to write, targets a specific piece of code, but can verify application-specific functionality. Static analysis requires no effort to write (beyond initial setup), targets the entire code base, but can only verify general code constructs. At least, that's the theory. How does it work in practice?

FindBugs supports a number of different ways of being used: command line, Swing GUI, integration into automated builds (i.e. via an Ant task and Hudson plugin), and Eclipse plugin. I decided to go with the plugin, and installation was as easy as adding the update site http://findbugs.cs.umd.edu/eclipse and installing it. Well, actually there were two gotchas. First, you need to be running Eclipse version 3.3 or greater for the plugin to work – RAD version 7 will not work. Second, you need to fully restart Eclipse after installing the plugin. I made the mistake of choosing the option to activate the plugin without doing a restart, which left portions of the plugin not working. It also seemed like you must bring up the Bug Explorer and Bug Details views, then restart Eclipse, in order to get those views working properly.

I used the FindBugs manual to get started. I selected my Time Reporter project to be the first guinea pig and ran FindBugs on it. I keep my code well-polished and well-tested with most Eclipse warnings turned on so I was not expecting FindBugs to turn up anything major. As I scanned the relatively small list of issues (under 30), I was surprised to see an actual defect! It actually took me a few moments of staring at it to find the problem. See the screen shot below.

It turned out this same defect occurred elsewhere in the code. FindBugs also identified cases of bad error handling that I would classify as defects: I was improperly ignoring return values from method calls like File.delete() or File.mkdir(). Most of these serious issues were in test code rather than application code, which made me feel a little better (but not much).

I fixed all the issues reported by FindBugs that I agreed with, turned off one FindBugs warning I did not agree with at all, and was then left with a small number of false positives – incorrect warnings about code that was actually correct. My preferred approach to compiler warnings is to have none in the code base. This is based on the fact that if your code base has existing warnings that should be ignored then it becomes very difficult to tell when you write some code that produces a warning that should instead be fixed. People become blind to all warnings if there are always some present. Warnings should be produced as part of the developer's regular process (i.e. writing code) rather than requiring an extra step. It was trivial to configure FindBugs to run automatically, but how to get rid of the unwanted warnings?

Eclipse warnings can be eliminated by using the Java 5 annotation @SuppressWarning, but FindBugs does not appear to support this annotation (or if it does I could not successfully determine what text must be supplied to the annotation to ignore the warning). I found some hints on the web that FindBugs does have the ability to ignore specific warnings, but it appears this feature has only been implemented in the Swing GUI and not in the Eclipse plugin. After further investigation I found a solution. I exported the current set of project warnings to an XML file, and then configured FindBugs to use that XML file as a baseline of warnings to ignore. This causes FindBugs to only report issues not in the baseline, leaving me with an empty list of FindBugs warnings – for now at least.

One annoying limitation of the FindBugs Eclipse plugin is that all configuration can only be done on a per-project basis. Unlike most other Eclipse functionality with global configuration that can be overridden on a per-project basis, FindBugs must be enabled and configured individually for each project. There were a few other wrinkles with the FindBugs Eclipse plugin. My general impression is that the plugin is still in a beta state, lagging behind the functionality offered by the FindBug Swing GUI.

Next I tried including FindBug as part of a continuous integration build running in Hudson. It was fairly easy to configure the Ant build to execute FindBugs on the project: the only snag was needing to allocate more memory for the JVM. Installing and configuring the FindBugs plugin for Hudson was likewise straight-forward, and resulted in a nice set of pages for viewing the trends and details regarding the FindBugs warnings. The big issue came when I wanted to filter out (exclude) all the warnings I did not want to fix. After much investigation and trial and error, I discovered that if I used the FindBug Swing GUI to create an exclude list, I could then configure the ant build to use that list as an exclude filter. Using the filter created by the Eclipse plugin did not seem to work. The FindBugs documentation concerning this was rather poor, but I did see a statement admitting that filter support was a bit messed up across the various FindBugs tools, and it sounds like this area will be targeted for improvement in the next year.

Despite these tooling problems, the ease with which FindBugs finds defects makes it definitely worthwhile to use on all Java projects. When I turned FindBugs loose against the code base of a reasonably-sized production application, it found 5 definite defects, 14 cases that were either potential defects or extremely bad coding style, and many other warnings (I didn't bother to count, but probably over 100) that were worth investigating and often worth fixing.

My conclusion from this is that using FindBugs is definitely worthwhile. I plan to roll it out to all my Java projects and integrate it into the automated builds so that the FindBugs results are also available from the continuous integration server. If you plan to adopt FindBugs then I recommend checking out some FindBugs tutorials. If you want to promote the use of FindBugs to your coworkers or management then I'll point out that FindBugs is a corporate standard at both Google and eBay and eBay reports that "using 2 developers to audit/review FindBugs warnings was 10 times more effective at finding P1 bugs than using two testers" (http://findbugs-tutorials.googlecode.com/files/UFIA-intro.pdf).

Based on my experience, I thought I had a good appreciation for what can go wrong. That changed recently when I experienced a day filled with too many problems and errors to be believed. The day started off innocently enough until a member of the production support team came by to inform us that he had accidentally terminated the database connection of one of our batch jobs due to transposing two numbers in the identifier. This by fluke matched our job instead of the one he wanted. Okay, no problem, we simply need to confirm that nothing was corrupted and restart the process. We checked our email for the notification email that is sent when a batch job abnormally terminates in order to verify which job had been affected. No such email was found. A little puzzled, we checked the server and confirmed that the process was no longer running. But another batch process was executing, and we identified it as a subsequent batch job dependent on the first. Subsequent jobs only run if the predecessors execute successfully, so we had a sinking feeling as we started checking the log files. Sure enough, due to a complete lack of error handling, the first job had reported a successful execution despite the database connection failure, which had caused the second job to start. That explained the lack of a notification email. The second job depended on the processing results of the first job, so the output of the second job was suspect and likely wrong. We had to kill the second job. If the first job had just failed, we could have restarted it without a problem, but now we had to investigate how to undo the effects of both jobs and manually restart the first.

Well, that didn't seem too bad, until I had time to think for a second. That is when I realized that our batch jobs are always scheduled at night or the weekends, and never during business hours. What was one doing running during the day? That prompted another investigation, which revealed that the job does normally runs on weekends. But the previous weekend there were problems with predecessors to this job that caused it to be delayed until one of the nights during the week. So why didn't it run at night? We were surprised to discover that it had – it just didn't finish. Due to performance issues, the job had run for over eight hours, extending into the day, before it was killed by mistake.

By late afternoon of that day we had multiple investigations underway trying to track down the various root causes of the problems we had identified. My mind reached the saturation point sometime in the afternoon, so I cannot remember the details concerning what was found. I suspect there were other problems unearthed that I have since forgotten. Nor where we able to get everything fixed that same day. That combination of problems, coming together on that one day, kept a surprising number of people busy for days sorting out the mess.

A famous example of root cause analysis is the presidential commission's inquiry into the 1986 US Challenger space shuttle explosion, particularly the observations of Nobel prize-winning physicist Richard Feynman. The basic finding of this investigation was that the explosion was caused by the failure of the O-ring seal in the right solid rocket motor due in part to unusually cold temperatures. They identified the problem, but did they find the root cause? The report does have a short section on the contributing cause of the accident.

Contrast this with Feynman's observations, which not once mention the O-ring seal and instead focus on deeper issues such as how NASA management evaluated shuttle reliability and safety. It was the solid rocket booster that failed, yet Feynman also investigated other major shuttle components. Feynman probed to the heart of the matter - the root cause - by not accepting limits on his why questions. He found that the engineers' warnings that it was too cold to safely launch went unheeded by management. His investigation was not without political consequences. Feynman's observations almost didn't make it into the final inquiry report - he had to fight to have it included - and it was relegated to an appendix. Feynman's final statement in his report elegantly summed up the root cause: "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."

My other example of root cause analysis comes from my own experience on a maintenance team for an operational business application. End users of this application had discovered bad data in one of the database tables in the production system. Other people on the team looked into the problem and determined that it was caused by a missing database trigger. Not missing as in forgotten to be added originally, but missing as in the trigger existed at some point in the past, but no longer did. When I learned of this situation, I started my root cause analysis by asking why the trigger had disappeared. Naturally I didn't get an answer, unless you count "I don't know". It was time to start investigating.

Triggers don't disappear by themselves. Someone had made a change to the database schema that eliminated the trigger. I doubted that someone would have explicitly deleted the trigger, if only because everyone was surprised it was gone. So it was a mistake - some other change that inadvertently eliminated the trigger. My chief suspect was a change to the underlying table. If the table was dropped and recreated for some modification, then dependent objects such as triggers would have been automatically dropped and would have needed to be recreated as part of the change. Of course, the database administrators (DBAs) on the team who make all the database changes know this. So why then would the trigger not be recreated?

I needed to find out how the DBAs normally performed table changes. A few questions later, I learned that the typical approach was to use their DBA tool to extract the DDL definition for the table and all related database objects (views, indexes, etc.), make the required changes to this DDL, then run it. I then tried this procedure for myself, selecting a table with a trigger. I used the DBA tool to extract the DDL. To my surprise, the resulting DDL did not include the trigger definition. This meant that I had found the probable root cause. While I didn't definitively know that this caused the problem, I knew it was very likely. More importantly, it was an issue that could be addressed to minimize the likelihood of triggers being dropped in the future. I notified the DBAs about my findings and this defect in the DBA tool was submitted to the company that developed it.

While I had a likely root cause, this didn't mean I was done with the root cause analysis. There were still more why questions to ask. Why wasn't this problem noticed sooner by the maintenance team before the change went into the production system? Was a proper review performed of the change? Why didn't system testing or user testing detect the missing trigger? Where any other triggers missing on production database tables? For each of these questions there is the potential for an answer that will identify how to help prevent this type of problem from reoccurring.

Root cause analysis is essentially a learning exercise, and as such should be a fundamental practice if your objective is perpetual learning or continuous improvement. It disappoints me to so often see it done poorly or not at all. The good news is that the ability to do root cause analysis can be trained and improved.

At its essence, root cause analysis involves asking "Why?" coupled with the determination to find answers that will help permanently resolve or at least improve the situation being dealt with. To start, you simply ask "Why did the problem happen?". If the answer tells you how to fix the problem but not how to prevent it in the future, then you need to keep asking why. Even once you start getting more profound answers, you can often continue asking why questions and learn still more.

Asking why is easy - figuring out the answer is hard. Both analytical and creative thinking skills play a role. At times you may feel like a detective, ferreting out clues like Sherlock Holmes. When faced with a question concerning a situation with no apparent answer, I've found it helpful to brainstorm ideas for what possibly could have led to the situation, and from there try and determine whether each possibility could have occurred.

Root cause analysis is not for the faint-at-heart. Asking probing questions and searching for answers about why things went wrong can make you unpopular, especially if your investigation involves other teams and other managers. You need to be careful that your efforts are not perceived as laying blame or trying to score political points by making others look bad. Instead, emphasize that it is a learning exercise whose purpose is to prevent future occurrences of the problem.

Next week I'll present some real-life examples of root cause analysis that touch on the above points. But you don't have to wait. You can start practicing root cause analysis today - just ask "Why?".

1. Initial investigation. My goal is to reproduce the reported problem in the application in my development environment. This may require obtaining more information from the user(s). If I can't reproduce it, I try to reason about possible causes to help me figure out what conditions are necessary to reproduce it. In hard cases, I may need to add extra error-checking or error-handling code to try to narrow down what is going on when the problem happens again.
2. Classify the problem. Many problems are defects: a flaw in the code or design that prevents the application from working as designed. Sometimes the user may be having troubles with the user interface - I classify this as a usability defect. Or the existing functionality may be too limited and the user is requesting something new - this I classify as an enhancement request. The distinction between defect, usability defect and enhancement is quite blurry. Often I will treat a usability defect like any other defect that needs to be fixed, but if such a defect requires significant user interface changes, then I classify it as an enhancement. If I determine that the reported problem is really an enhancement request, then I am finished with my defect fixing process (enhancements are typically handled through a different process).
3. Analyze the cause. My goal is to write an automated unit test that proves the existence of the defect by failing - i.e. I'd expect the test to pass if the defect didn't exist. I may need to do some analysis to determine what section of code is failing, and may need to iterate on this a few times if the defect is especially hard to pin down. For certain problems, including most usability defects, it is too difficult to write an effective test so I rely on manual testing and debugging.
4. Fix the defect. Now that I know the cause of the defect, I can fix it. This is where having a test for the defect really pays off, since I can run the tests after making my fix and verify that the problem is corrected by having the tests pass.
5. Learn from the defect. The defect may be fixed, but I'm not done. Each defect is a learning opportunity; it represents a failure in something I've done that I can correct and improve. So I ask myself questions like the following. Why or how was this defect introduced into the code? Was it an error in design, a copy-and-paste error, a failure to consider special conditions like a null return value, improperly understood library, etc.? Why didn't the unit tests catch this problem? Why didn't reviews or manual testing catch this problem? Would similar defects exist elsewhere in the code, due to the same or a related failure that caused this problem? What can I do to prevent this type of defect from happening again?
6. Act on my learning. Based on what I've learned from this defect, I take action to improve myself, the application, and the team's processes for the future. Possible actions include writing more unit tests for the problematic section of code, refactoring a difficult-to-use API to eliminate the problem that occurred, informing other developers of why this defect happened, and improving error handling to make it easier to reproduce similar defects in the future.