Shakespeare quotes aside, every software development project has to make decisions on how many defects to fix and which ones to leave alone prior to shipping. While I have seldom seen this question debated within projects, the advice from industry thought leaders varies considerably. The Agile and Lean methods of software development in particular have somewhat opposing perspectives.

I believe that considering both sides of this question provides a fuller understanding of the issues and better equips us to answer appropriately. Therefore in the two sections below I explore the reasons behind both sides of the debate.

To Fix

• Shipping poor quality, defect-ridden code can upset users, turn away customers, and lead to a hard-to-shake bad reputation.
• The decision that a feature is worth developing is made with the expectation that it will work correctly. So any defects found in a feature means that the feature is still incomplete until these issues are fixed.
• Defects provide feedback regarding the development process. Each defect represents an opportunity to do a root cause analysis of what led to the defect and put countermeasures in place to prevent re-occurrence. The Lean mindset of "Stop the line" demands that new development be put on hold to fix newly discovered defects.
• Defects introduce the risk of compounding quality problems. The impact of a defect can be more significant than initially realized. Defects can be inadvertently replicated in other parts of the system. Enhancing components with too many defects can slow progress to a halt, as the system becomes essentially a shifting quicksand that is too unstable to work on. Constantly fixing defects helps maintain a high velocity of development over time.
• To mitigate risks in not fixing defects, each defect needs to be analyzed to understand its impact, cause, and required changes to fix. But after performing this analysis most of the work is usually done - the fix is relatively straightforward. Waiting to decide later to fix the defect (e.g. in a subsequent release) causes all the knowledge gained in the analysis to decay over time which is wasteful (in the Lean sense).

Not To Fix

• Significantly delaying the release of software to fix all defects leads to a loss of immediate revenue and potentially loss of market share due to competitors beating you to market. So you cannot afford to wait to fix all defects.
• The entrepreneurial mindset, especially for startups, is to ship early to get feedback from paying customers. Perfection is the enemy of the good.
• Under at least some versions of Scrum, defects are considered new tasks that are added to the product backlog to be prioritized by the product owner. This prioritization is based on the defect's impact (severity and likelihood of occurrence) and the effort required to fix it. Many minor defects will therefore likely never be fixed as new functionality will typically be of higher value.
• Stopping to analyze and fix defects disrupts developers who are in the middle of working on other functionality and is wasteful.
• Fixing defects in functionality that is already otherwise finished development and testing will require additional regression testing. Not fixing now and waiting until enhancements to this functionality are needed minimizes the extra effort required.

Conclusion

Shakespeare was wrong. There is actually a third perspective regarding whether or not to fix defects: avoid the question as much as possible by focusing on defect prevention. The Lean mindset of building quality in avoids all the waste associated with finding, analyzing, and fixing defects and should be our preferred approach. Only when it fails and the occasional defect is introduced do we then have to answer the question.

• Documentation is more effective than code at communicating high-level or cross-cutting design and operational concerns.
• Larger organizations or distributed organizations cannot rely on face-to-face communications or having everyone co-located in one room so documentation has a role to play in knowledge transfer and communication.
• Documentation can target a non-developer audience, such as business representatives.
• Documentation helps protect against team turnover, which while a bad practice is not uncommon when using a vendor for development or maintenance or using separate development and maintenance teams.
• The act of creating documentation helps clarify thinking and identify gaps, thus functioning as a form of quality control.

This last reason is actually quite significant as it is often under-appreciated. In my experience it has happened quite often when I am working on design documentation for a body of code that I identify things that are sub-optimal, such as a badly named class or an unwanted dependency between components. This form of quality control is most valuable, however, when it uncovers gaps such as missing functionality or flawed design. Such gaps are very difficult for most other forms of quality control like testing or reviews to find.

The use of documentation templates makes it much easier to find these gaps by acting essentially as checklists of items to consider. I recently came across a great set of comprehensive templates covering all aspects of software development by Klariti . For a team or organization the price of these templates is ridiculously low - the full set of software development templates costs far less than a day's salary - and there are an assortment of free templates as well.

There are some potential drawbacks to avoid when using documentation templates. Some people have a tendency to want to fill in every section of a template or to use all available templates. This can waste a lot of time and effort. Focusing on identifying what is relevant and useful to document and doing only that is much more effective. Another drawback is that while the templates are in Office format (Word or Excel), you might be better served using a different medium such as a Wiki or a text-based format that is more friendly to version control. In these cases I would convert the templates to the desired medium.

Even if your team or organization has some templates, I think it would be beneficial to check out Klariti's templates and use them as a checklist to see if there is anything missing or needing revision within your own. That link again is: Klariti's software development templates.

Too often I see projects and even entire organizations misaligned with one or more of these principles and in the worst cases taking actions in complete violation of these principles. The result is predictable: poor quality or significantly extra cost to achieve adequate quality, with the complete absence of high quality.

Achieving high quality is difficult. It suffers from what I call the weakest link problem: no matter how many things you do well, it is the area you are poorest at that generally dictates the overall quality. Another way of putting it is this: there are many ways to fail at developing high quality software and only a few ways to succeed.

So to help you achieve high quality here is my list of the top seven quality principles in software development. For each principle think about the implications - how might your organization be acting in ways that contradict it? How might you change to better align with the principle?

People as individuals and teams are the most significant factor affecting quality

Software engineering research has consistently found that variations between people are by a large margin the single largest contributing factor to quality and productivity. While good process is important, good people have a much greater impact.

Achieving high quality requires people capable of doing high quality work, both individually and as teams. While individual excellence is important, most software is produced by a team of people and it is the team's overall quality of workmanship that ultimately determines the quality of the software being produced.

Within organizations, throughout the blogosphere, and even in many software development books too much attention is placed on process and methodology and too little on the quality of the people doing the work.

All software has defects

Any significant piece of software has defects, no matter how carefully it has been produced. Zero defects is an Utopian ideal, but not a realistic goal. With extreme discipline and effort you can get very close to zero defects, as evidenced by the group working on the space shuttle control software, but under more normal conditions defects are unfortunately a fact of life.

It is impossible to fully test software

For even the most trivial of functionality like adding two numbers there are a nearly infinite set of tests that could be performed. So completely testing a significant piece of software is impossible. Testers therefore must appropriately choose and perform an extremely small subset of all possible tests.

Defects are more expensive to fix the later they are found

As time elapses from the point at which a defect is introduced, the effort required to fix the defect grows. There are several reasons for this. The most significant is that as the software moves through subsequent phases of the software development life-cycle - from development into test and then finally into production use - the effort involved to get to that phase increases. Work previously done, like testing and deployments, has to be repeated to some degree. Another reason is that over time, the developer's memory of the problematic functionality in question fades, thus requiring more effort to recover the context.

Build quality in

This principle comes from the lean thinking literature and addresses the issue highlighted by the prior principle. Finding and fixing defects is considered waste - non-value-add work - and thus according to lean should be minimized. This is accomplished in a twofold manner: first, use practices that help prevent the introduction of defects, such as specifications with examples, and second use practices such as test driven development and code reviews that find defects as quickly and cheaply as possible after they are added.

Adopt your approach to quality based on the level of criticality and complexity

Different contexts require different actions. Criticality and complexity are the two most significant factors to consider in order to determine the level of quality you require and the approach you will take to achieve it. Criticality is the measure of how important the application is to the business and/or the users and is generally assessed using categories such as life critical, mission critical, business important, and casual use. Complexity is the measure of how difficult it is to understand and work with the code. A number of factors increase complexity such as complicated algorithms, sheer volume of code, and poor architecture resulting in high coupling.

Higher quality requires more quality-focused activities and better execution

In order to achieve a higher level of quality corresponding to a lower volume of production defects requires that you introduce less defects and/or find and fix more defects throughout your software development process. This process can be modeled as a series of activities acting as feedback loops or filters that each prevent or remove some percentage of defects.

Using this model it becomes obvious that the only way to reduce the defects remaining at the end of the process is to either make existing activities more effective - what I call better execution - or add more activities to filter out additional defects. The choice of which quality-focused activities to use and the effort to put into each requires deliberate planning which I have written more about in my article titled Filter by Failure Mode Matrix: A Method for Planning Quality.

Role-Based Attitudes

I will first answer this question on a role by role basis, starting with the role of tester. Since one of the primary objectives and professional skills of testers is to find defects, I would expect them to generally be happy when they find problems. The more significant (e.g. critical) the defect or more tricky to find, the happier I would expect them to be. If the defects are trivially easy to find, or are so serious that they prevent further testing, then I could see testers getting upset because they are prevented from exercising their craft to the fullest level of their capabilities.

What about developers? When developers find defects in other people's code, perhaps via a code review, this is fairly similar to the tester role above so I would expect developers should generally be happy in this scenario.

When a developer finds a defect in their own code as part of the development process - e.g. by having an automated test that unexpected fails after a code change then how should they feel? It is only natural to feel some disappointment at having made a mistake, but I do not believe this is the best reaction. In fact, it may be harmful. If you experience mild distress or pain whenever you find your own defects, you may subconsciously start to try to avoid the pain by, for example, doing less testing. So I believe a better reaction is to be happy when you find your own defects. This may seem challenging to do, but consider these rationale:

• Finding the defect validates your skill at performing whatever defect detection activity you were doing - whether it was automated testing or personal code review - and reinforces the value of performing that activity.
• Discovering this defect provides a learning opportunity on how you might better prevent or detect this type of defect in the future. Learning opportunities are good.
• As a professional, you want to pass along high quality code. Every defect you find in your own code is one less defect others will find, thus raising the quality of your final output.

The final scenario to consider is when someone else finds defects in the developer's code. A typical example is when the developer considers a feature finished and a tester is testing it. Now this must be bad, right? Even more than the prior scenario it seems natural to feel disappointment. Each defect that is discovered is essentially downgrading the quality of what you have produced. However, some of the reasons we listed previously for feeling positive still apply. In particular if the defect was still found within the team rather than being passed along to the end user or client, then this is a good thing. It can be hard to see this as a developer when it is your code at fault.

So let us consider the final role: that of the team lead or manager who is supervising activity rather than specifically doing coding or testing. This role has no personal stake in individual defects, but instead is typically concerned about the broader implications to the project or product. Assuming that high quality is a key objective, how should such individuals feel about defects? One tendency is to react negatively because of the extra effort and schedule time required to fix defects. Finding too many defects can also lead to alarm bells over the level of quality. I feel that since the team lead or manager essentially represents the whole team, their reaction should correspond to the perspective of the entire team.

Team-Based Attitudes

So what should the attitude of the team as a whole be to discovering defects? To properly answer this question we need to know what the team's objective is. I am going to assume it is to build working software that is being used and meeting users' needs. The existence of defects is clearly bad since they threaten this objective. But what about the discovery of defects? Discovering a defect means the team gains more information about the state of the software which it will typically use to improve the software by fixing the defect, and which the team can also use as feedback to improve.

Discovered defects exist prior to being found, but the team does not know about them until the time of discovery. This means that the team should experience simultaneous conflicting reactions: happy that the defect was discovered, yet unhappy that the defect exists.

Individual Attitudes

So what should the attitude of the various individuals on a team be towards defects? Should it be based on their role? I believe that in an ideal team every individual will understand and adopt the common objectives of the team. This means that everyone's attitude towards defects should ideally be the same, no matter what the role.

As I have observed, in practice attitudes tend to be quite far from this ideal state. People maintain their own personal objectives and viewpoint in addition to or instead of the team's. Often they do not appreciate the distinction between discovering defects and the existence of defects. To correct this I believe team members need to be clear on the team's objectives and understand how the existence and discovery of defects impacts these objectives. I hope this article will help in that regard.

(Did you stop and think?)

In a philosophical sense, I could argue that testing is never truly done. Especially if testing is defined using James Bach's broad definition of questioning a product in order to evaluate it, I could easily create examples of testing happening days or weeks after the examination of the product has finished. Even ignoring this broad definition and using a narrower definition of executing software to verify that results match expectations, there are still an essentially infinite number of possible tests that can be performed on even the most trivial of software, so there is no way to ever be fully done. These philosophical points, however, do not match the intent of my question, which is really about the allocation of effort towards testing. When should you, as a developer or tester, stop putting effort towards testing a piece of software and consider it done? This is essentially elaborating upon a definition of done for testing.

My answer to this question is based upon the specific goals or objectives for the testing being performed. In my experience there are usually two primary goals to testing:

1. Find defects.
2. Assess whether the software is ready to be promoted / released.

Testing can have other goals - for a fuller discussion, see the article What is a Good Test Case? (pdf) by Cem Kaner.

So the simplistic but essential answer to my question is that testing is done when its objectives have been achieved. More specifically, you are done testing when:

1. You are unlikely to find additional defects.
2. You have a sufficiently high level of confidence that the software is ready to be promoted / released.

These answers are rather brief and do not provide much in the way of actionable guidance, so I expand on them below. Before I do so, however, I must address another factor: project constraints of budget, schedule, and resource availability. If you run out of budget or time, you generally need to stop testing. This does not mean, however, that testing should be considered done. On the other hand, testing for an indefinite period of time oblivious to cost or duration does not seem appropriate. Ideally a balanced approach is taken. The primary testing goals are modified to incorporate the notion of employing a reasonable level of effort (cost) and duration given the desired quality level. For example life critical software demands much higher quality and thus much greater effort towards testing than software intended for casual, personal use.

Defining Done: Finding Defects

If your goal in testing is to find defects, then ideally you should stop testing after you have found all the defects. Unfortunately, there are several flaws with this theory:

• Testing is very unlikely to find certain types of defects. Even the combination of several different styles of testing is unlikely to find more than 75% of the total defects. Each individual type of testing, even when carefully executed with a high degree of skill, is unlikely to find more than 50% of the defects.
• Testing is relatively inefficient at finding defects, even assuming you could find all of them. You will potentially spend a very, very long time trying to find all the defects, especially the last few.
• You have no way of knowing in advance if all the defects have been found, or whether more remain in the system.

So in practice you need to abandon the idea of finding all the defects and use a different approach. Instead, evaluate the likelihood of finding more defects based on the effort you have already put in and based on the results you have obtained so far. If this likelihood is too low then you are done testing: additional testing would not provide a sufficient return on investment in terms of new defects found compared to the effort expended. Some points to consider when making this evaluation:

• If you have just found a defect, this is a signal to keep testing. It may seem counter-intuitive, but in general the more defects you find, the more likely it is that there are additional defects.
• If you have only exercised a small portion of the overall functionality and already found defects, then this is a signal to continue testing.
• If you have been testing a particular piece of functionality for a while and are not finding new defects, then this is a signal for you to stop testing.
• If you are struggling to come up with new tests that are meaningful, then this is a signal that you are done. Another sign of this is when the defects you do find are found to have low relevance to users and the decision is made not to fix them.
• If the tests you are performing are becoming more and more complicated and taking significantly more effort, but you are only occasionally finding defects, then this is a signal to stop.
• When testing a larger set of functionality like an entire application, the question of whether to stop testing can and should be applied at the level of individual features or components. One reason for this is that defects tend to cluster. This commonly leads to a system having a few error-prone components that account for up to 80% of the total number of defects, while other components can be significantly higher quality. Once you have identified a component that appears error-prone, focusing additional testing on it is quite likely to find more defects.
• When testing a large system, you should revisit the decision to stop testing a particular area as you gain more information. For example, imagine you have three features of a system to start testing for the first time. You start testing the first feature, and initially find a bunch of defects without much work, but shortly afterwards the defects seem to become much harder to find. At this point, it is best to stop testing feature one and switch to feature two. Feature two you find no defects in at all, so after a shorter period of time you switch to feature three. Here you find the occasional defect every so often, but eventually find it hard to create meaningful tests. Since it seemed like feature one had the highest defect density, you return to it to test, but after a short while and only one new defect found you stop again. At this point you realize that feature two has received the least testing, and there are parts of its functionality you have not tried, so you return to test it, and do find one defect, but no more so you stop. Testing feature two happens to give you some ideas for new meaningful tests to try for feature three so you return to testing it, but find no new defects there either, so at this point you decide you are done testing all three features.

Defining Done: Assessing readiness

If your goal in testing is to assess whether the software is ready to be promoted to the next level of testing or released into production, then you are done testing once you have obtained a sufficiently high level of confidence in your assessment. If your assessment is a 'no-go' - do not proceed with the promotion / release - then it is likely that issues you found will need to fixed and trigger additional testing in order to obtain your 'go' recommendation. (This assumes, of course, that there is sufficient budget and schedule for additional testing and also assumes that the relevant decision maker(s) care about quality and pay attention to your assessment. If this is not the case, then you perhaps should not make assessing readiness a testing goal.)

Factors to consider in performing this assessment are:

• What level of quality is required? Is the system life-critical, mission-critical, or only for casual personal use? This quality level relates directly to the level of confidence you need to have that the system is ready.
• It is far easier to determine that the system is not ready. Finding a critical defect (aka a showstopper), or finding even a few serious defects in critical functionality is usually sufficient to warrant a no-go assessment. In contrast, achieving sufficient confidence that the system is good for release usually takes more work.
• How much of the system's functionality have you tested? If there are significant features that are mostly or entirely not tested, then you likely will not be prepared to recommend it is good to go. So frequently you should favor testing critical functionality broadly across the entire application versus focusing in detail on a single component. This conflicts to some degree with the approach suggested for the prior goal of finding defects where you might choose to focus your testing on an error-prone module.
• False confidence is a very real danger. This is especially true for developers - I have seen or heard of many who have an unrealistically high level of confidence that their code works, in some cases without any testing whatsoever. As a specific example, I have heard developers state that if their code compiles, it is good enough to be promoted to acceptance testing. (And this was for business-critical applications, not casual use.) This is one reason to have on development teams one or more testers with a testing mindset who will ensure for themselves that the system will work rather than assume it. To combat this false confidence it may help to read my article Would you trust your life to your code?.
• Your testing should be designed as much as possible to find defects of the highest severity and highest relevance to users, as these are the kinds of defects most likely to warrant a no-go assessment. You might be able to find lots of cosmetic defects like spelling mistakes in the application's help documentation, but this does not really contribute much towards an assessment of readiness. You may have noticed that even when discussing the prior goal of finding defects, my assumption is that you will bias your efforts towards finding more severe and more relevant defects.

My goal for this article was to motivate you to think more carefully about the question of when testing is done and enable you to be more effective as a tester. To test :) whether I achieved this goal, please leave a comment below letting me know what you thought of the article. Thanks!

• Measuring Defect Potentials and Defect Removal (pdf)
• Software Assessments, Benchmarks, and Best Practices
• Estimating Software Costs: Bringing Realism to Estimating

The information I provide in the remainder of this article comes primarily from these references.

Estimating Software Size

The first step in Caper's approach is to estimate the size of the software. His preferred size metric is function points, which are a language/technology neutral evaluation of the business functionality based primarily on an assessment of program inputs, program outputs, and data storage. Another common metric is logical lines of code, often abbreviated as SLOC. KLOC represents 1000 lines of code.

Lines of code are a convenient metric in that they can be measured automatically with a tool, whereas function points required a trained function point counter. Nevertheless, function points are superiour in several key ways:

• Roughly fifty percent of defects are due to problems with requirements or design rather than coding, for which metrics in terms of lines of code do not make much sense. In particular, different implementations of the same functionality can have significant variances in the lines of code required by up to a factor of four. A longer implementation will have fewer requirement or design defects per KLOC and thus artificially seem to have higher quality, when in reality the code is simply bloated. (This is related to the issue of measuring developer productivity by lines of code produced.)
• The use of multiple languages complicates source code counts, and this is far more common than one may expect. Even a simple web application typically includes JavaScript, HTML, CSS, and perhaps SQL in addition to the primary language (e.g. Java). Function points are language-independent.
• Function points can be determined once the design is known - lines of code require waiting till coding be completed. This allows function points to be useful for planning and estimation purposes much earlier.

Since Caper prefers function points, the defect metrics he provides in his writings are typically expressed using this measure. While I understood the reasons, I found these metrics difficult to apply because I had no idea of what the function point counts were of the software I was working on. So I was pleased to discover that benchmarks such as this one exist to convert between function points and lines of code for various languages. One function point on average is roughly 50 logical lines of Java code. I use this conversion factor in the sections below.

Defect Potential

All software has the potential of having defects. Defect potential is a measurement of the expected number of defects in a particular piece of software. This is also called the injection rate - the number of defects being introduced throughout development. The primary factor determining the number of defects is the size of the application in function points. The maturity - experience, skill, and attention to quality - of the development team is another key factor in determining defect potential. The following table specifies how to calculate the expected number of defects given these two factors.

Some key defect prevention activities contributing to reduced defect potential are:

• Close customer collaboration during requirements / design (e.g. JAD sessions)
• Prototyping
• Feedback / learning from design and code reviews

The above metrics assume a linear relationship between defects and size, but as a system gets larger there are typicaly more interactions between pieces and more complexity, and thus a greater likelihood of defects than a linear increase would suggest. For very large systems, a more accurate metric is as follows: number of defects = function points raised to an exponent. For average organizations, use 1.25 as the exponent. (Good organizations can lower this to 1.15, while poor-performing organizations have this elevated to 1.35.)

Defects can be categorized by origin - the type of activity that produced the defect. The table below shows this breakdown for average organizations.

Defect Removal

Defect removal is the identification and elimination of defects after they are introduced. The cumulative defect removal rate or defect removal efficiency of a development project is calculated as the number of defects eliminated prior to the release to production divided by the total number of defects found after 90 days of production use.

The following table shows how the defect removal rate varies with the maturity level of the team, just like defect potential, and shows the expected number of post-release defects based on the defect potential and removal metrics.

Removal efficiency varies for defects of different origins, as the following table shows using statistics for average organizations.

Quality control procedures such as testing and reviews (inspections) vary in their effectiveness at removing defects as illustrated in the following table.

Peak defect removal rates for a given activity are typically obtained only through the use of skilled, experienced staff who take a rigourous, disciplined approach to performing the activity. As an example consider unit testing. As it is normally performed it has a 25% defect removal rate. But an experienced developer following test-driven development will achieve nearly 100% code coverage and typically write better tests, leading to a higher 50% defect removal rate.

Predicting Defect Levels

The overall or cumulative defect removal rate for a develoment effort can be calculated by aggregating together the individual defect removal rates of the quality control procedures used by the team as follows: Cumulative removal rate = 1 - the product across all procedures of (1 - individual removal rate per procedure).

For example, if a team uses only unit testing (25% removal), functional testing (30%), and regression testing (20%), then the cumulative rate = 1 - (1-0.25) * (1-0.30) * (1-0.20) = 0.58 or 58%.

The overall number of defects in production (or UAT) can be calculated using the defect potential of the team to determine the expected number of defects introduced, and using the cumulative defect removal rate to determine the number of defects remaining. Approximately 25% of defects will be high severity.

For example, an average organization injecting 1 defect per 10 lines of code for a 25 KLOC application will end up introducing a total of 2500 defects. Given a cumulative defect removal rate of 95%, this means that 2375 defects will be found, leaving 125 defects remaining of which 31 can be expected to be high severity.

In the past my general pattern when doing code reviews has generally been to focus on production code and do a more minimalistic review of the unit test code. But over the past year, I have been seeing indications that this might be a mistake. One respected colleague even advised me to reverse my priorities and focus primarily on the tests. So why would reviewing test code be so important?

Reasons to review test code

Research by Caper Jones indicates that errors in test cases are quite common - as high as 20%. In other words, one in five test cases has something wrong. Caper was likely referring to manual test scripts rather than automated test code, but there's no reason to expect the act of automation to magically fix all such errors. I recently reviewed a set of unit tests for a class and found one test that made no sense - it appeared to be verifying behavior against the wrong expected result. When I raised the issue with the developer, they confirmed that the test was wrong. Fixing the test revealed a defect in the production code which they then promptly fixed. This was the same production code that I had already carefully reviewed.

Automated tests can have other issues besides simply being incorrect. Tests can be incomplete, weak, fragile, or hard to read and maintain.

Incomplete tests fail to test certain functionality. Reviewing the results of a code coverage tool is a very easy way to identify gaps in coverage of the existing production code. A more careful review of the actual tests is needed, however, to ensure that all required functionality is actually implemented since code coverage cannot identify gaps when production code is missing.

Weak tests have a low likelihood of detecting defects. One example is tests for trivial getters and setters - such tests are so weak that I generally consider them useless clutter and delete them when I see them in a review. Test weakness is really a continuum from virtually useless, as per my example above, to extremely powerful. Part of reviewing tests is to identify when the power of the test can be improved. A while ago I had to change a business rule to use less-than-or-equals logic rather than less-than logic as originally specified. I confirmed there were tests covering this rule, then changed the logic and re-ran the tests expecting them to fail, but to my surprise they all passed. Upon investigating, I discovered that the tests improperly exercised the boundary condition - they tested a value below the boundary and above the boundary, but not on the boundary. So the equal-to-boundary case, which was the logic I was changing, was not tested. This prompted me to review the tests for other similar functionality, where I found the same pattern of weak boundary tests, a few of which had led to defects escaping detection. If we had been more diligent reviewing the unit tests when this functionality was first written we could have avoided these problems.

Fragile tests have a high likelihood of requiring updates when production code is changed. In other words, changes are likely to cause such tests to start failing. Having too many fragile tests can lead to maintenance issues down the road - developers may choose to delete such failing fragile tests rather than fixing them, or worse might choose to stop running the tests altogether. Martin Fowler termed this test cancer. Reviews by individuals experienced in maintaining unit tests are the only way I know of detecting and minimizing fragile tests.

Test code needs to be clean - easy to read and maintain - just like production code. In fact, many proponents of automated tests argue that well-written tests can function as documentation highlighting what the production code is supposed to do and providing examples of how to use specific APIs. In my experience, using tests as documentation only works when the test code is clean, which generally requires reviews to ensure it happens consistently.

Incorrect perceptions regarding reviewing test code

Given all these compelling arguments for reviewing unit tests, why have I not reviewed tests as much in the past? The 'excuses' I have used seem to be due to incorrect perceptions on my part.

As an architect, I often do what I call architect-level reviews, focusing on the overall structure of the production code and non-functional attributes. I am often not familiar with the detailed functional requirements the code is trying to satisfy. So I don't bother looking at the details of the test code, since I usually can't confirm if the expected results are actually what the requirements specify. But wait - if I am able to still effectively review the production code, then I can perform the same type of review for the test code.

I focus first on production code due to my underlying belief that correct and clean production code is more important than test code. In one sense, this is true. But there are other viewpoints. One view is that if it isn't tested, it doesn't work. Another view is that the long-term vitality of the production code over multiple release depends on a solid, well-written suite of automated tests.

Related to the prior error in perception is the issue of schedule and/or budget constraints. Given limited time or effort, where should I allocate my time for a review? If my priority is production code, then I focus on that and neglect the test code. But this very question contains a flawed assumption: it assumes I should sacrifice the thoroughness of the review for other constraints. But the defects I fail to detect as a result of this sacrifice will likely get caught and fixed later at much greater expense. Reviews are one of the most effective means of defect removal - they are usually twice as effective as most kinds of testing at finding defects and are half as costly to fix the defects that are found.

Conclusion

While I hope this article helps convince you of the importance of reviewing test code, I wrote it just as much for myself to reinforce the importance of doing these reviews. Here are some pointers for getting started:

• Use a code coverage tool like Cobertura to review how well the tests cover the production code.
• Review the tests first, before reviewing the production code.
• Use the following questions as a guide to reviewing test code:
  • Are the requirements fully tested? Are there any omissions of functionality?
  • Are the tests weak or fragile?
  • Are the tests readable and the code clean (e.g. no duplication)?

One approach for doing this selection is a method I call the Filter by Failure Mode Matrix. The basic idea behind the matrix is to identify the activities and practices that will act as filters to either prevent or mitigate quality-related failure modes in the software development process. This is inspired by failure mode and effects analysis except it is applied to the development process itself rather than a piece of software. The categorization of failure modes in the matrix should be as broad as possible to minimize the size of the matrix, while remaining specific enough to effectively identify specific activities and practices that will be effective at preventing or mitigating each failure mode. Many of the failure modes correspond to defect root cause categories.

The matrix defines two sets of filters for each failure mode: primary filters and secondary filters. Primary filters consist of those activities or practices that the team will rely on as their first line of defense in preventing or mitigating the corresponding failure mode. Secondary filters act as a second line of defense for those issues that make it through the primary activities. The activities and practices assigned as a secondary filter are either less effective in preventing or mitigating that particular failure mode, cost more to perform in terms of budget or schedule compared to the primary filters, or are performed later in the overall development process compared to the primary activities. When the primary filters consist solely of activities performed by developers, then the secondary filter should include activities performed by non-developers in order to serve as an independent assessment of quality with respect to that particular failure mode. This is especially important for mission-critical and life-critical systems.

Multiple activities can be listed as primary filters or as secondary filters. The intent is not to require only a single activity per filter category. In fact, to obtain higher quality research tells us that multiple quality-enhancing activities are a necessity.

While most failure modes have general applicability, some are more contextually-dependent so you should ensure that the list of failure modes you use is meaningful to the particular software development effort you are undertaking.

Both the relevant failure modes and the effectiveness or applicability of quality-enhancing activities and practices can vary significantly based on the nature of the application functionality. For example, web-based user interface screens will experience usability defects and automated functional testing is more difficult to apply. In contrast a scheduled batch process does not need to worry about usability defects and automated functional testing is usually much easier to apply. So when a system combines multiple components that differ like this, this can be handled in one of two ways. First, a single matrix can be used with additional failure modes representing the two different components. Based on our above example, you could have a failure mode for web-based functionality defects and a second category for batch process-based functionality defects. This allows the latter category to specify automated functional tests as a primary filter. The second approach is to define separate matrices. This should only be done when there are significant differences between the matrices.

While a filter by failure mode matrix can be defined up front, you should expect to evolve it over time. Examples of how a matrix can evolve are:

• A particular activity is found to be ineffective and is replaced with another activity in the matrix.
• A new failure mode is identified based on defects that occur in user acceptance test or production. A new row is added for this failure mode with corresponding filters identified.
• The overall quality level is discovered to be too low, so additional activities and practices are added throughout the matrix.

The following table provides a sample filter by failure mode matrix.

So what logical argument can be made to determine who is responsible for quality? I will start with an obvious possible answer: the quality assurance department. Perhaps they are responsible for quality because that is what their name means: they assure the organization that good quality work is being done. Oh wait, can they always do that? What if the work being done is poor quality? Can the QA department prevent or fix that? The answer is no. So the best QA can do in such cases is assure the organization that the quality is bad. Well, if QA cannot assure good quality, perhaps we can still hold them responsible for the bad quality work. And maybe QA would be responsible, to a limited degree, if their approach to assessing quality or their policies contributed to the bad quality work not being prevented or not detected sooner. But shouldn't the responsibility lie with those doing the actual work?

So it must be the developers who are primarily responsible for quality. They write the code, so they are responsible if it has defects. This sounds correct at first glance, but to be sure I will examine some different categories of defects. Logic errors - yup, developers are responsible. Misunderstood requirements - again let's make the developers responsible - they should have asked for clarification. Ambiguous requirements - this starts getting into a gray area - perhaps the business analysts drafting the requirements could have been clearer. Contradictory requirements - the developers are off the hook for this one - the business analysts have to take responsibility here. So for most aspects of quality the developers are responsible, and for other aspects it is the business analysts. I know - let's just make the development team as a whole responsible for quality. Great, case closed!

Wait, what's that? You are saying that I missed a category of defects - the category of missing requirements? Well, that might be due to poor analysis on the part of the business analysts, but ultimate responsibility falls on the source of the requirements - the product owner / business team. They're not usually considered direct members of the development team, though, so making the development team solely responsible for quality no longer seems like the right answer. And what about those in management responsible for setting organizational constraints on the development team - constraints like schedule, budget, or resources? If these constraints are too severe they are guaranteed to lead to quality problems, either in the form of defects due to sacrificing quality, or in the form of insufficient functionality due to sacrificing scope. So shouldn't these managers also be held responsible, to some degree, for quality?

Everywhere I look, at every unit within the enterprise associated with software development, each has a role to play in contributing to the success of a software development effort. In other words, each is providing a service. Imagine, if you would, one of these units with absolutely no responsibility for providing good quality. This irresponsible unit can provide the worst quality service, and the other units would just have to deal with it, either by completely avoiding the irresponsible unit's service, or by accepting responsibility for addressing the bad quality aspects to bring it up to the required level of quality. In either case, the other units are essentially doing the job of this irresponsible unit, which means it no longer has a rational reason for existence within the organization, and can be eliminated. Thus, in a proof by contradiction, I have demonstrated that a unit without any responsibility for quality should not exist within a rational organization. (I am well aware that most enterprises are only rational in theory, not in practice, and such irresponsible organizational units can exist in reality, but I believe my basic logical reasoning is not tarnished by this.)

This then leads back to my original question. Who is responsible for quality? The answer is everyone.

You must be careful, however, that the cost of these defect prevention practices does not become excessive. That would introduce a different type of waste – non-value adding process. The waterfall method of software development is an example of this. One of the principles behind waterfall is that careful requirements analysis and design will minimize downstream defects during coding and testing. Put another way, it is a good idea to understand what you need to build before you start building it. The problems with waterfall arise from going to extremes in applying this principle. Requirements analysis is done up front for the entire project as a big batch based on the theory that it minimizes rework due to future change, but in reality the constant pace of requirement changes plus the learning that occurs throughout the project will result in increasing amounts of change the longer the time spent doing requirements. In contrast, Scrum and Kanban apply this principle using a balanced approach – project level requirements are done at a high level, and the more detailed analysis is done on individual user stories just prior to implementing them. (See for example the article Scrum and CMMI – Going from Good to Great: are you ready-ready to be done-done?.)

In order to effectively adopt a defect prevention practice two pieces of information are needed:

1. Specific, actionable steps to apply the practice.
2. The expected benefit. What categories of defects does the practice intend to prevent? This helps determine when to apply the practice and helps to evaluate it after adoption to assess its effectiveness.

If we consider the idea of careful requirements analysis and design mentioned above as a prevention practice, the benefits are fairly clear - prevent requirement and design based errors - but specific actionable steps are missing so it does not qualify. (In fact, this is one of the contributing factors why waterfall projects can end up in the analysis paralysis anti-pattern.)

Now that the groundwork has been laid I can present some specific defect prevention practices. This is not a comprehensive list – many other practices are possible. The practices I have chosen to discuss are ones that I have used and am confident that they work.

Use Understood Methods Rule

The basic formulation of the rule is quite simple: when coding a method only invoke other methods whose behavior you clearly understand and are confident will work the way you want. I have written a separate article providing specific guidance on how to apply this rule.

This practice generally aims to prevent interface errors - which I define generally as defects between two separate pieces of code. Research suggests that a significant proportion of defects are due to these kinds of errors (See for example the paper An Empirical Study of Software Interface Faults.)

I find this rule particularly valuable when applied to the invocation of methods between classes and especially between components. In this case it helps prevent integration errors which are usually not caught by unit testing.

Design by Contract

The idea of design by contract is to precisely specify the behavior of methods to help ensure that they are invoked correctly by callers and that the callers receive the results they are expecting. This is done by precisely specifying the preconditions and postconditions of methods. The chief proponent of design by contract is Bertrand Meyer, whose book Object-Oriented Software Construction is a classic on the topic.

Method preconditions are conditions that must be true in order for the method to successfully execute and fulfill its postconditions. Preconditions are most commonly applied to method arguments. For example, a method to convert a string to a date might have the precondition that the string argument must not be null. Preconditions can also be applied to the state within the class or even external state. For example, a method to delete a particular object from the database might have the precondition that the object exists within the database.

Method postconditions are conditions that the method guarantees to be true after execution, assuming the preconditions are met. Postconditions are most commonly applied to the return value of methods, but like preconditions can also be applied to the state within the class or external state. Returning to the example of a method that converts a string to a date, such a method could have two postconditions. First, that the method will return a corresponding date object that is not null if the input is a string in a valid date format, and second that the method will throw a specified exception if the input does not correspond to a valid date format.

The combination of preconditions and postconditions forms in essence a contract between the method and its caller. The caller promises to fulfill the preconditions in exchange for the method guaranteeing that the postconditions will be met.

Despite the fact that the name of this practice contains the word "design", this approach does not require a separate up-front design of each method. The goal is to have a clear specification of behavior once the method is finished – how you arrive at it is not important to this practice. I tend to start with an initial idea for a method’s contract that I evolve as I write tests and implement the method’s logic using test-driven development.

There are several options for specifying pre- and post- conditions. Some teams rely solely on their automated unit tests to serve as the specification, but I prefer a more concise specification provided as part of the method definition. In Java I typically use JavaDoc to document pre- and post- conditions and programmatically check argument preconditions at the start of the method. I typically formally specify pre- and post- conditions only on methods that are intended for use by other classes or components. In Java, this is typically public and protected methods of interfaces and classes.

This practice is very closely related to the Use Understood Methods Rule, and they go hand-in-hand. Knowing a method’s pre- and post- conditions is necessary to fully understanding it. As I stated above, I tend to only apply formal design by contract to methods intended for use outside the class in question, which means this practice is really aimed at preventing integration defects.

Defensive Coding

Defensive coding is named after the practice of defensive driving and is based on the same mindset of expecting problems to occur and actively taking precautions to avoid them. Defensive coding is applied by adopting a language-specific set of idioms that minimize or prevent common coding errors when using the language. These idioms are often reflected in coding standards.

Here are some examples of defensive coding idioms for Java:

• When comparing if a variable is equal to a constant, put the constant first. This avoids a potential null-pointer exception (if the variable is null) by invoking the equals() method on the constant, which is never null.

  public boolean isAdmin(String userId) {
    String constant = "admin";
    return constant.equals(userId); // Instead of userId.equals(constant)
  }
  

• Always use braces to define a block of code for an if, else, while, for, or do statement, even if the block contains only a single line of code. This avoids the problem of later adding a second line of code indented to the same level as the first and mistakenly thinking it will invoked as part of the block.

  public void addOptions(String userId) {
    if (isAdmin(userId)) {
      addAdminOptions();
    } else {
      addRegularUserOptions();
    }
  }
  

• Use the Java 5 for-each construct rather than using a loop index variable to manually iterate through a list. This avoids the problem of having an off-by-one error in constructing the loop.

  public void processOptions(List options) {
    for (Option option : options) {
      option.process();
    }
  }
  

Defensive coding aims to minimize coding errors, both at the time of coding and in the future when the code is being modified by others. While these types of errors are typically easily detected by unit testing, I find that using these idioms (after the initial adoption) takes virtually no effort or thought on my part, making them literally a no-brainer to use.

Example-Based Requirements

The idea behind this practice is to express requirements as much as possible in terms of concrete examples rather than the generalized wording typically used in use cases and lists of business rules which is almost always ambiguous. I have written a separate article providing further details on example-based requirements which includes a specific example. :)

The practice of example-based requirements aims at minimizing requirement errors, particularly errors due to misunderstanding or misinterpreting. The examples should also be used as acceptance test cases, in which case they help detect design or coding errors (although unit tests should identify most of these first).

Conclusion

I encourage you to choose one or more of these practices to adopt in your current development work. There will be extra effort initially to understand and become comfortable with a given practice, but this will decline over time as you achieve mastery of it.