This chain of thought then prompted me to ask myself this same question. What do I find most disturbing when reviewing code? My instinctive reaction was to answer "defects", but upon a little reflection I realized this was not true - I expect to find defects as a fundamental principle of quality. So it usually does not bother me to find a few during a review.

There are times when I am very disappointed when reviewing code - what am I finding at those times? Here are some specific occurrences:

• Code riddled with defects reflecting a fundamental lack of understanding about the requirements.
• Code very difficult to understand due to poor names and overly complicated logic that seemed repetitive or unnecessary.
• A large amount of non-GUI code written without any supporting tests.
• Code with inconsistent formatting and style.

What is the common theme here? After further reflection, I realized that the common element underlying these situations that I find most disturbing is a lack of professionalism / craftsmanship. This can manifest in a number of ways as indicated by the above list. The key criteria is whether a developer is helping achieve or is hindering our mission as software developers, based on what they produce for code. My evaluation of what is most disturbing is at its essence based on my core values and beliefs concerning software development.

In the past my general pattern when doing code reviews has generally been to focus on production code and do a more minimalistic review of the unit test code. But over the past year, I have been seeing indications that this might be a mistake. One respected colleague even advised me to reverse my priorities and focus primarily on the tests. So why would reviewing test code be so important?

Reasons to review test code

Research by Caper Jones indicates that errors in test cases are quite common - as high as 20%. In other words, one in five test cases has something wrong. Caper was likely referring to manual test scripts rather than automated test code, but there's no reason to expect the act of automation to magically fix all such errors. I recently reviewed a set of unit tests for a class and found one test that made no sense - it appeared to be verifying behavior against the wrong expected result. When I raised the issue with the developer, they confirmed that the test was wrong. Fixing the test revealed a defect in the production code which they then promptly fixed. This was the same production code that I had already carefully reviewed.

Automated tests can have other issues besides simply being incorrect. Tests can be incomplete, weak, fragile, or hard to read and maintain.

Incomplete tests fail to test certain functionality. Reviewing the results of a code coverage tool is a very easy way to identify gaps in coverage of the existing production code. A more careful review of the actual tests is needed, however, to ensure that all required functionality is actually implemented since code coverage cannot identify gaps when production code is missing.

Weak tests have a low likelihood of detecting defects. One example is tests for trivial getters and setters - such tests are so weak that I generally consider them useless clutter and delete them when I see them in a review. Test weakness is really a continuum from virtually useless, as per my example above, to extremely powerful. Part of reviewing tests is to identify when the power of the test can be improved. A while ago I had to change a business rule to use less-than-or-equals logic rather than less-than logic as originally specified. I confirmed there were tests covering this rule, then changed the logic and re-ran the tests expecting them to fail, but to my surprise they all passed. Upon investigating, I discovered that the tests improperly exercised the boundary condition - they tested a value below the boundary and above the boundary, but not on the boundary. So the equal-to-boundary case, which was the logic I was changing, was not tested. This prompted me to review the tests for other similar functionality, where I found the same pattern of weak boundary tests, a few of which had led to defects escaping detection. If we had been more diligent reviewing the unit tests when this functionality was first written we could have avoided these problems.

Fragile tests have a high likelihood of requiring updates when production code is changed. In other words, changes are likely to cause such tests to start failing. Having too many fragile tests can lead to maintenance issues down the road - developers may choose to delete such failing fragile tests rather than fixing them, or worse might choose to stop running the tests altogether. Martin Fowler termed this test cancer. Reviews by individuals experienced in maintaining unit tests are the only way I know of detecting and minimizing fragile tests.

Test code needs to be clean - easy to read and maintain - just like production code. In fact, many proponents of automated tests argue that well-written tests can function as documentation highlighting what the production code is supposed to do and providing examples of how to use specific APIs. In my experience, using tests as documentation only works when the test code is clean, which generally requires reviews to ensure it happens consistently.

Incorrect perceptions regarding reviewing test code

Given all these compelling arguments for reviewing unit tests, why have I not reviewed tests as much in the past? The 'excuses' I have used seem to be due to incorrect perceptions on my part.

As an architect, I often do what I call architect-level reviews, focusing on the overall structure of the production code and non-functional attributes. I am often not familiar with the detailed functional requirements the code is trying to satisfy. So I don't bother looking at the details of the test code, since I usually can't confirm if the expected results are actually what the requirements specify. But wait - if I am able to still effectively review the production code, then I can perform the same type of review for the test code.

I focus first on production code due to my underlying belief that correct and clean production code is more important than test code. In one sense, this is true. But there are other viewpoints. One view is that if it isn't tested, it doesn't work. Another view is that the long-term vitality of the production code over multiple release depends on a solid, well-written suite of automated tests.

Related to the prior error in perception is the issue of schedule and/or budget constraints. Given limited time or effort, where should I allocate my time for a review? If my priority is production code, then I focus on that and neglect the test code. But this very question contains a flawed assumption: it assumes I should sacrifice the thoroughness of the review for other constraints. But the defects I fail to detect as a result of this sacrifice will likely get caught and fixed later at much greater expense. Reviews are one of the most effective means of defect removal - they are usually twice as effective as most kinds of testing at finding defects and are half as costly to fix the defects that are found.

Conclusion

While I hope this article helps convince you of the importance of reviewing test code, I wrote it just as much for myself to reinforce the importance of doing these reviews. Here are some pointers for getting started:

• Use a code coverage tool like Cobertura to review how well the tests cover the production code.
• Review the tests first, before reviewing the production code.
• Use the following questions as a guide to reviewing test code:
  • Are the requirements fully tested? Are there any omissions of functionality?
  • Are the tests weak or fragile?
  • Are the tests readable and the code clean (e.g. no duplication)?

• A higher percentage of defects are found when using reviews – as high as twice as many than testing.
• When an issue is detected by a review, the root cause of the problem is usually known. Defects found by testing require further work to analyze.
• Reviews can be performed earlier in the development life-cycle than testing.
• Reviews provide more precise and more immediate feedback for developers to learn and improve from than testing.

Steve McConnell in his book Rapid Development summarizes these findings on pages 73 to 75.

Given these many benefits, it is disappointing that more teams do not use code reviews. If you are part of a team that does regular code reviews, then I congratulate you. Just because you do code reviews, however, does not guarantee that you will see all the benefits. In fact, I have often observed highly ineffective reviews. One example from my own experience is when a developer was asked to do a review but viewed the task as an annoying interruption from more interesting work. They did a very quick and superficial 'review', and then reported that everything looked fine. As a task the review was completed, but the expected quality improvements were not achieved. In a sense the review did not really happen.

How can you avoid this from happening to you and your team? In this article I present a set of strategies for achieving effective code reviews. These are based on my years of experience performing code reviews and figuring out what works the best. Some of these ideas are inspired by concepts from PhotoReading, a multi-step system for efficiently extracting desired information from written text (aka reading, but not as it is traditionally done).

Adopt a Critical yet Playful Attitude

The primary purpose of a code review is to critique the code - to find defects, potential problems, or simply poorly-written sections – which requires a critical mindset that is different from the typical problem-solving mindset used by developers as they solve code. You cannot simply read over a section of code and accept what you see: you need to challenge the assumptions and decisions behind it. This is especially important for finding errors of omission. It is much harder to find a defect because of what was not written, and to do so requires you have a questioning, suspicious attitude towards the code you are reviewing. You need to mentally prepare yourself before starting a review by reminding yourself of the purpose of the review and the need for a critical, questioning attitude.

Maintaining this critical attitude throughout your review, however, can be difficult, especially if it is not your regular mode of thinking. It helps to cultivate a delight or enjoyment in finding issues with the code, like you are playing a game searching for treasures hidden by an opponent. Unearthing a serious defect should feel equivalent to hitting a home run in baseball. Viewing the review as a fun yet challenging game where you score points for the issues you find will boost your motivation and keep you focused in that critical mindset.

The critical attitude required for reviews is similar to the questioning mindset needed for root cause analysis. In root cause analysis, however, you are trying to determine why something failed, whereas in a code review you are trying to find all potential causes of failure.

Take Multiple Passes Through the Code

In order to do an effective review you must learn how the code is structured, what it is trying to accomplish, and where it falls short. It is difficult to do this all at once as you read through the code for the first time. A much more effective approach is to use multiple passes to review the code with the initial passes focused on learning about how the code works and later passes focused on critiquing the code.

The number of passes you will need will depend on the size of the code base you are reviewing, its complexity, and how thorough a review you are doing. As a starting point, here is a suggested approach for reviewing a code base in multiple passes.

1. Problem Overview: Review the requirements or specification to understand what the code is supposed to do. What problem is it trying to address? What requirements is it implementing?
2. Solution Overview: Review the design document or specification to learn the overall solution to the problem. How are the requirements being addressed?
3. High Level Organization: Review the code or detailed design document to figure out how the code is organized and functions at a high level. What are the packages and the major classes within each? I find UML class diagrams showing class relationships and hierarchies very useful for this pass, and if none already exist as part of the documentation I often end up sketching out such a diagram. UML sequence or state diagrams can also be useful to highlight the main interactions or behavior of the system.
4. Major Class Review: Review the code for the most important classes as identified previously. While issues and problems can be found in earlier passes through the code, particularly at the design level, this pass marks the transition from learning about the code to critically analyzing it for problems.
5. Multiple Detailed Review Passes: The previous passes will have given you a sufficient understanding of the code base to critically review the remainder of the code base. Even at this point, however, it is still valuable to use multiple passes. I explain why in the next section.

Focus on One Goal Per Pass

One common response to the problem of poor-quality reviews is the creation of a review checklist that lists all the issues a reviewer should be looking for. Often these checklists enumerate many specific items and go on for more than a page. Two examples of checklist items are: (1) checking the return codes of system calls, and (2) using prepared SQL instead of SQL literals for parameters. While the idea of checklists has merit, the reviewer is given no guidance as to how to effectively use the checklist. Short-term memory, which can only hold approximately seven pieces of information, is all needed for analyzing the code. Trying to remember a page of issues to look for while reviewing simply does not work.

The solution to this is to focus on a single goal for each detailed review pass through the code. Each goal can be as specific as an individual checklist item, but I find this level of detail is seldom necessary and leads to too many passes. The goals I prefer to use generally correspond to categories of issues. For example, instead of a checklist item such as checking return codes of system calls, I have the goal of reviewing error handling. As I make my review pass with this goal in mind, I can ignore the bulk of the logic and focus instead on how errors might occur and how they are handled.

Unlike the long checklist, this solution is workable because you only need to hold a single piece of information – the current review goal – in your short-term memory while you are reviewing. Focusing on a single goal, however, provides an additional benefit: it actually improves our ability to find problems in the code pertaining to the goal. How does this work? Our eyes and other senses constantly transmit a wealth of information to our brain – too much information, in fact, to fully process in detail. The brain must therefore selectively filter the information it receives. The brain decides what information to select based in part on our goals and desires – what we view as important. In essence, our thoughts and perspective affects how we view the world. When we focus on a single goal such as error handling, as we review the code the brain is more likely to focus on segments of code dealing with error handling (or the lack thereof). This allows you to review the code faster: you can skim along, skipping past sections that do not apply, then slow down to carefully analyze a relevant block or a suspicious section, and then speed back up again. Changing your goal between review passes causes your perspective to shift, allowing you to identify a different set of issues in the same code base. This greatly increases the effectiveness of the review by catching more issues across a wider variety of categories.

Here are some typical goals I use for my reviews:

• Functionality: Does the code meet the business (functional) requirements? Is the logic correct?
• Class Design: Does each class have low coupling and high cohesion? Is the class too large or too complicated?
• Code Style: Is duplication of code avoided? Are any methods too long? Are typical coding idioms / standards followed?
• Naming: Are packages, classes, methods and fields given meaningful and consistent names?
• Error Handling: How are errors dealt with? Does the code check for any error conditions that can occur?
• Security: Does the code require any special permissions to execute outside the norm? Does the code contain any security holes?
• Unit Tests: Are there automated unit tests providing adequate coverage of the code base? Are these unit tests well-written?

Add Review Comments Directly to the Code

How should issues found during the review be reported back to the original developer? One commonly used approach is to create a review document listing all the issues found. For each issue the problematic section of code must be referenced, usually by supplying the file name and line number. Management often prefers the review document because it can be used for process auditing and generating metrics. Exclusive use of a review document for recording issues, however, causes problems for the reviewer and the original developer. For the reviewer, switching from the code to a separate document can easily break their flow, especially since they must record not just the issue, but also the reference back to the code. For the original developer, once they start fixing issues in a particular file, the references (file name and line number) of other issues in the same file can quickly become out of date, especially if refactorings such as move method, extract method, or rename class are performed.

I believe, therefore, that the most effective approach to recording issues is to add them directly to the code base as comments, at the place the issue occurs. Each such comment should be tagged with a special label – I use REVIEW – so that the issues can be easily found later by the original developer. In fact, if you use the Eclipse IDE for Java development, you can define a task tag for this type of comment that will cause all such comments to appear in the Task view. Once the review is completed, the code with the review comments can be checked back into the version control system to be made available to the original developers. If you must produce a review document to comply with your organization's processes, then I would recommend creating the document at the end of the review from the review comments you added to the code.

Summary

Code reviews are a powerful practice for improving software quality. The strategies I have found personally useful in increasing the effectiveness of my reviews are:

• Adopt a critical yet playful attitude.
• Take multiple passes through the code.
• Focus on one goal per pass.
• Add review comments directly to the code.

If you have any additional strategies you have found particularly useful, please let me know.

Based on the code bases I have seen and worked with, polishing code is not a frequent practice. In fact, I have encountered people who are surprised by my commitment to it, or who do not believe in its value. Many of these people are not developers, or have not spent much time doing ongoing software maintenance. Part of the reason for the lack of appreciation is that polishing code provides long term benefits for a short term cost. This induces project managers and developers on development projects with compressed time frames to skip such kinds of activities. In other cases, non-technical managers simply have no idea of the poor quality of code their developers are producing. One project manager I dealt with personally reviewed and made minor grammatical corrections to the documentation produced by the project team, but had no similar processes in place for the poor quality code produced by the project team.

So why do I believe that polishing code is valuable? I have several reasons. The most important is that it minimizes the long term cost of maintaining the software. To explain this, I like to use the concept of technical debt. Quality issues such as poor design in software represent a debt for which you will need to make continuing interest payments later in the form of increased effort to understand or change the software. This is the case even if the software functions correctly. To pay off that debt, you must address the underlying quality issues - in other words, you must polish the code. Martin Fowler wrote a recent article about this topic in the context of good design that explains the trade-offs involved. Another reason I polish my code is that it is a permanent record of my capabilities as a professional software developer. My reputation amongst developers who need to work with my code later is in large part determined by my code's quality. Code of mine that is difficult to understand or change or is riddled with defects will have maintenance developers cursing at me. I would rather have them be pleasantly surprised at finding helpful comments and a clean design.

So now you hopefully believe that polishing code is valuable, but you may be asking how exactly to go about it. In terms of timing, polishing of code should ideally be done throughout the development of a section of code. In reality, however, it is much easier to focus on one issue at a time, and the primary issue is just getting the code to work. Therefore after the code is written and passing the automated unit tests, it is only natural for it to have some rough spots. This is the ideal point to do a review of your code to identify places to polish. My basic goals when polishing are to improve the understandability of the code and to improve its design. During the review, I look for specific problems impacting these two goals. When I find such a problem, I immediately rectify it by refactoring the code. Often it is as easy as using one of the built-in refactoring tools of the development environment to safely make the change with a near-zero risk of introducing defects.

Some examples of specific problems I look for are:

• Duplication of logic in two or more places.
• Undesired dependencies. For applications divided into layers such as persistence, business and user interface, the dependencies between the layers need to be managed. Finding undesired dependencies such as the use of persistence code directly within the user interface layer is a strong indicator of a design problem.
• Overly complex logic, like a large switch statement or series of nested if-else statements. Sometimes such logic can be drastically simplified through the use of object-oriented design principles.
• Too many classes within one package.
• Too many methods within a class, particularly public methods. This is often an indicator that the class is trying to do too much and should be split into multiple classes.
• Overly long file, in excess of a couple thousand lines. Even in Java where each public class is forced to be in its own file, it is possible to end up with very long files. This may be because there is too much logic within the class, or it may be because of many nested classes or private classes in the same file. In the latter case, these can be moved to one or more separate files.
• Overly long methods. Anything longer than about one screen warrants a closer look to see if it can be simplified, usually via an extract method refactoring.
• Poorly named classes, methods, fields or local variables. One indicator I look for is a one-sentence comment describing a vaguely named variable or method. Usually the variable or method can be renamed to capture the essence of the comment and the comment can then be deleted.
• Lack of a class comment. A concise summary of the purpose of the class is often quite useful. If it is difficult to come up with such a summary, it is an indicator that the class may be trying to do too much or does not have a clear focus.

Ron Jeffries' site http://www.xprogramming.com/ has a great series of articles that provide concrete examples of polishing code, including before and after samples of the source code.

Polishing your own code is good, but it is better if your entire team adopts the practice. An effective way to ensure code is polished within a team is via code reviews. While ideally everyone should polish their own code, having a separate reviewer check for code quality problems is a big help. This is especially true for ensuring that the code is understandable by someone other than the original developer.

Applications with polished code are easier to understand and modify, which helps extend their useful life. Applications with poor quality code are more likely to be partially rewritten or even completely replaced if the cost of ongoing maintenance becomes too high. Which of these applications would you prefer to contribute to? The choice is yours.