There are a number of pragmatic practices that address this objective while avoiding the pitfalls afflicting formal traceability. I will start with a simpler form of traceability and then move on to practices seemingly unrelated to traceability.

Specification Highlighting to Track Test Coverage

A tester given a written specification to verify the software against must have some way of keeping track of their progress. One simple method is to highlight each statement in the specification as they test it. This is essentially tracking test coverage of the specification. I credit James Bach for this idea.

Testing Dashboard

A testing dashboard is great at illustrating overall testing progress and can be used as part of the summary in the final test report. To produce the dashboard first divided the system under test into test areas that usually correspond to features, components, or significant non-functional requirements. Then create a grid with the test areas as rows, and the columns report testing progress for each area. Key information to report per area is an assessment of quality (is this area ready to ship?), and the thoroughness of testing (test coverage and test effort). For an example and fuller explanation see James Bach's presentation on testing dashboards.

The testing dashboard is closest in form to a requirements traceability matrix - there's still a grid and something like requirements are listed down the grid. The big difference is that the dashboard shows a summary of testing for each area rather than listing individual tests and showing how they map.

Agile User Stories with Acceptance Tests

A common approach used by Agile methods is to divide requirements into fine-grained increments of business value called user stories and then define with examples or automated tests the criteria by which to accept that the story was correctly implemented. This takes a two-pronged approach to address traceability's objective. First, using fine-grained increments of functionality that are quickly developed minimizes the amount of work-in-progress that needs to be tracked and thus reduces the chance of mistakes. Second, the process produces tests for each user story early, sometimes before coding starts, so missing tests for a requirement is much less likely. Agile basically sufficiently changes how development is done to negate virtually all the sources of problems that would otherwise justify a requirements traceability matrix.

Task Tracking via Task Board

Popularized by Kanban but also used by many Agile teams, a task board is a practice for tracking the status of tasks. The board has multiple columns representing different status: the minimum set is usually "Not Started", "In Progress", and "Done". Tasks are placed on the board according to their status, and the team meets daily to discuss the tasks and update appropriately. I vastly prefer a physical task board on a wall, but for teams not co-located many online versions exist. Tasks are often quite fine-grained - one user story is often decomposed into multiple tasks.

Many teams include a status of "Testing" on the board, or use a symbol on the card to signify the completion of testing. Other quality control procedures such as code reviews can also be tracked. See the image below for an example of a task board with such states.

The combination of fine-grained tasks, daily updates, and tracking quality ensures that each requirement that is tackled by the team is properly developed. This achieves traceability's objective.

Conclusion

Despite formal traceability being too effort-intensive for too little value the underlying objective should not be ignored. I have described a variety of alternative practices that help ensure that requirements are properly developed and tested with a more modest level of effort.

Formal Traceability

The most frequent incarnation of formal traceability I see is the requirements traceability matrix (RTM). This is a two-dimensional table, often represented by a spreadsheet, in which the one dimension lists requirements and the other dimension lists the artifact being traced to - business objectives, test cases, or design documentation. Cells within the table are filled in with a mark when the corresponding artifact and requirement are related.

The example traceability matrix below lists requirements across as columns and test cases down as rows. Notice that requirement 2.2 has no corresponding related test case, which is considered a traceability gap that indicates that this requirement has not been tested.

Traceability Troubles

Below are listed the aspects of traceability that trouble me.

• I have successfully developed and enhanced high quality software applications that have pleased clients and been used in production without needing formal traceability. So does it really provide value? Traceability seems like it addresses the problem of having unimplemented or untested requirements. This has rarely been a problem on the development projects I have been on. We are much more likely to struggle with ambiguous or unclear requirements, missed or new requirements, and incorrect logic in obscure special conditions.
• The tracing of requirements to test cases assumes they can be identified and listed which in turn limits the testing approaches that can be used. The context-driven school of testing would challenge these limitations. Exploratory testing in particular does not fit the confines imposed by a formal requirements traceability matrix.
• While the value of producing the initial RTM is unclear to me at best, I am even more dubious about the value of maintaining a RTM over time. The typical claims is that for for future requirement changes (e.g. enhancements) the RTM can be used to do impact analysis and determine what tests need to be executed and/or changed. Regarding impact analysis, the RTM might get you started, but the analysis must always consider code-level dependencies not reflected in the RTM (through, for example, reuse of common functionality), and must consider other requirements and functionality that are not directly related (and thus not reflected in the RTM) but make assumptions that are no longer true under the proposed change. The idea that the RTM can be used to determine what tests to executes makes several poor assumptions. First, that the tests are manual and not automated. If you have an automated test suite, you can just rerun the whole suite to identify failing tests - you never need to worry about running a subset. Second, that reusing manual tests is a good idea. If a test is worth re-executing often, it is frequently worth automating. For those tests that are manual, much of the benefit of having human testers is lost if they simply re-run the same scripts over and over, since there is a low probability of such re-executions failing and testers are less likely to find defects when they do occur. A better approach is to let the tester come up with their own variations each time to try to creatively break the system.
• Tracing requirements to design is conceptually flawed. It at best very indirectly addresses the objective of traceability - ensuring the customer gets the functionality they require. Doing this mapping is in reality quite hard because the mapping from design to code (and from requirements to design) is not a strict one-to-one and is often a complex many-to-many, especially for holistic non-functional requirements like performance or security. Furthermore, the idea of tracing to design assumes that there are fine-grained pieces of design that can be identified and referenced. This presumes a certain approach to design and its documentation that is at odds with many modern methods of software development.
• For small-scale software applications, the system is simple enough that producing a RTM is a reasonably small effort. The number of requirements and test cases is relatively small. But the chance of errors occurring that would be caught be traceability is likewise low because the system's complexity is low. For very large systems with hundreds or thousands of requirements and tests, it is far more conceivable that requirements may be forgotten to be implemented or tested. But while the benefits of formal traceability are higher, the efforts are much, much higher. For a RTM, the number of cells to complete essentially scales non-linearly with the number of requirements in the system.
• I do not understand the obsession with testing in the context of traceability and especially RTMs. Reviews or inspections are much more effective at finding defects than testing. So why do I never hear of formal traceability from requirements to reviews?

Conclusion

Software engineering researchers and industry leaders have reached the same conclusions about traceability. Andrew Kannenberg and Dr. Hossein Saiedian in Why Software Requirements Traceability Remains a Challenge (PDF) are clearly big proponents of requirements traceability, but look at their conclusions:

• "Unfortunately, manual traceability methods are not suitable for the needs of the software engineering industry."
• "Currently existing COTS traceability tools are not adequate for the needs of the software engineering industry."

Robert Glass in his book Facts and Fallacies of Software Engineering points out the underlying cause of troubles implementing traceability: "when a requirement commonly links to 50 or more design requirements and each of those links to some very much larger number of coding elements and coding elements may be reused to satisfy more than one requirement, we get a burgeoning complexity problem that has resisted manual solution and even tended to thwart most automated solutions" (page 78)

I hope the above discussion has convinced you to tread cautiously when it comes to traceability. In a future article I will look at alternatives to formal traceability that achieve the same underlying objectives.

A common problem I encounter from the use of these generalized requirement statements is ambiguity. This has multiple negative impacts. The user representatives providing requirements and the business analysts documenting them can both review the same written requirement and agree that it is correct, based on their discussions, while actually holding differing interpretations. But it gets worse. The developers implementing the requirement who typically do not have the benefit of the interactions and discussions with user representatives can arrive at an even larger range of interpretations. And if you have a separate tester, who knows what they might come up with.

To illustrate the problem, here are some questions about the example business rule provided above demonstrating the amount of ambiguity in just one single sentence:

• By the phrase "last transaction" do you mean the last transaction chronologically? If so, which transaction date should be used to determine this – the date of posting or the date of acceptance?
• The phrase "more than a year ago" could be based on calendar year or could be based on the corporate fiscal year. Which is it?
• Assuming the phrase "more than a year ago" is calculated based on calendar year, how exactly should the calculation work? It could be based solely on the year – e.g. warn if the year of the transaction is two or more years less than the current year. It could be based on date – e.g. warn if the transaction date happens before the same month and day as today last year. Or it could be based on a number of days – e.g. warn if the transaction date is more than 365 days before the current date. (Note that the last two scenarios will produce different results in a leap year.)

The solution to the problem of ambiguity is to use examples. And not just a few examples here and there to support particularly complex requirements – requirements should be largely based on examples, with the generalized description simply providing the higher level intent or summary. Examples should ideally be created by user representatives, or at the very least carefully reviewed by them. The benefit of examples is not just to bring greater clarity to the development team but also to ensure the users are clear on what the system will do. This helps avoid unpleasant surprises when the users first see and play with the working system.

The Agile community has figured this out a long time ago, thus prompting the move to very short user stories supplemented with acceptance tests instead of formally elaborated use cases. The acceptance tests function as the specific concrete examples. The user story only provides a high level summary in part because of the expectation that further face-to-face conversation will take place regarding the requirement.

If your environment requires more formalized requirements documentation then keep using use cases and lists of business rules, but keep the written text focused on the higher-level intent while examples provide the specifics. To provide an example :) of this, here is the example business rule rewritten to fit this new approach:

As the account manager I want to receive a warning when the account has no recent activity.
Examples:

• Most recent transaction in account based on posting date of May 1, 2009 with current date May 1, 2010 produces the warning "No recent activity in account".
• Most recent transaction in account based on posting date of May 2, 2009 with current date May 1, 2010 does not produce a warning.

You must be careful, however, that the cost of these defect prevention practices does not become excessive. That would introduce a different type of waste – non-value adding process. The waterfall method of software development is an example of this. One of the principles behind waterfall is that careful requirements analysis and design will minimize downstream defects during coding and testing. Put another way, it is a good idea to understand what you need to build before you start building it. The problems with waterfall arise from going to extremes in applying this principle. Requirements analysis is done up front for the entire project as a big batch based on the theory that it minimizes rework due to future change, but in reality the constant pace of requirement changes plus the learning that occurs throughout the project will result in increasing amounts of change the longer the time spent doing requirements. In contrast, Scrum and Kanban apply this principle using a balanced approach – project level requirements are done at a high level, and the more detailed analysis is done on individual user stories just prior to implementing them. (See for example the article Scrum and CMMI – Going from Good to Great: are you ready-ready to be done-done?.)

In order to effectively adopt a defect prevention practice two pieces of information are needed:

1. Specific, actionable steps to apply the practice.
2. The expected benefit. What categories of defects does the practice intend to prevent? This helps determine when to apply the practice and helps to evaluate it after adoption to assess its effectiveness.

If we consider the idea of careful requirements analysis and design mentioned above as a prevention practice, the benefits are fairly clear - prevent requirement and design based errors - but specific actionable steps are missing so it does not qualify. (In fact, this is one of the contributing factors why waterfall projects can end up in the analysis paralysis anti-pattern.)

Now that the groundwork has been laid I can present some specific defect prevention practices. This is not a comprehensive list – many other practices are possible. The practices I have chosen to discuss are ones that I have used and am confident that they work.

Use Understood Methods Rule

The basic formulation of the rule is quite simple: when coding a method only invoke other methods whose behavior you clearly understand and are confident will work the way you want. I have written a separate article providing specific guidance on how to apply this rule.

This practice generally aims to prevent interface errors - which I define generally as defects between two separate pieces of code. Research suggests that a significant proportion of defects are due to these kinds of errors (See for example the paper An Empirical Study of Software Interface Faults.)

I find this rule particularly valuable when applied to the invocation of methods between classes and especially between components. In this case it helps prevent integration errors which are usually not caught by unit testing.

Design by Contract

The idea of design by contract is to precisely specify the behavior of methods to help ensure that they are invoked correctly by callers and that the callers receive the results they are expecting. This is done by precisely specifying the preconditions and postconditions of methods. The chief proponent of design by contract is Bertrand Meyer, whose book Object-Oriented Software Construction is a classic on the topic.

Method preconditions are conditions that must be true in order for the method to successfully execute and fulfill its postconditions. Preconditions are most commonly applied to method arguments. For example, a method to convert a string to a date might have the precondition that the string argument must not be null. Preconditions can also be applied to the state within the class or even external state. For example, a method to delete a particular object from the database might have the precondition that the object exists within the database.

Method postconditions are conditions that the method guarantees to be true after execution, assuming the preconditions are met. Postconditions are most commonly applied to the return value of methods, but like preconditions can also be applied to the state within the class or external state. Returning to the example of a method that converts a string to a date, such a method could have two postconditions. First, that the method will return a corresponding date object that is not null if the input is a string in a valid date format, and second that the method will throw a specified exception if the input does not correspond to a valid date format.

The combination of preconditions and postconditions forms in essence a contract between the method and its caller. The caller promises to fulfill the preconditions in exchange for the method guaranteeing that the postconditions will be met.

Despite the fact that the name of this practice contains the word "design", this approach does not require a separate up-front design of each method. The goal is to have a clear specification of behavior once the method is finished – how you arrive at it is not important to this practice. I tend to start with an initial idea for a method’s contract that I evolve as I write tests and implement the method’s logic using test-driven development.

There are several options for specifying pre- and post- conditions. Some teams rely solely on their automated unit tests to serve as the specification, but I prefer a more concise specification provided as part of the method definition. In Java I typically use JavaDoc to document pre- and post- conditions and programmatically check argument preconditions at the start of the method. I typically formally specify pre- and post- conditions only on methods that are intended for use by other classes or components. In Java, this is typically public and protected methods of interfaces and classes.

This practice is very closely related to the Use Understood Methods Rule, and they go hand-in-hand. Knowing a method’s pre- and post- conditions is necessary to fully understanding it. As I stated above, I tend to only apply formal design by contract to methods intended for use outside the class in question, which means this practice is really aimed at preventing integration defects.

Defensive Coding

Defensive coding is named after the practice of defensive driving and is based on the same mindset of expecting problems to occur and actively taking precautions to avoid them. Defensive coding is applied by adopting a language-specific set of idioms that minimize or prevent common coding errors when using the language. These idioms are often reflected in coding standards.

Here are some examples of defensive coding idioms for Java:

• When comparing if a variable is equal to a constant, put the constant first. This avoids a potential null-pointer exception (if the variable is null) by invoking the equals() method on the constant, which is never null.

  public boolean isAdmin(String userId) {
    String constant = "admin";
    return constant.equals(userId); // Instead of userId.equals(constant)
  }
  

• Always use braces to define a block of code for an if, else, while, for, or do statement, even if the block contains only a single line of code. This avoids the problem of later adding a second line of code indented to the same level as the first and mistakenly thinking it will invoked as part of the block.

  public void addOptions(String userId) {
    if (isAdmin(userId)) {
      addAdminOptions();
    } else {
      addRegularUserOptions();
    }
  }
  

• Use the Java 5 for-each construct rather than using a loop index variable to manually iterate through a list. This avoids the problem of having an off-by-one error in constructing the loop.

  public void processOptions(List options) {
    for (Option option : options) {
      option.process();
    }
  }
  

Defensive coding aims to minimize coding errors, both at the time of coding and in the future when the code is being modified by others. While these types of errors are typically easily detected by unit testing, I find that using these idioms (after the initial adoption) takes virtually no effort or thought on my part, making them literally a no-brainer to use.

Example-Based Requirements

The idea behind this practice is to express requirements as much as possible in terms of concrete examples rather than the generalized wording typically used in use cases and lists of business rules which is almost always ambiguous. I have written a separate article providing further details on example-based requirements which includes a specific example. :)

The practice of example-based requirements aims at minimizing requirement errors, particularly errors due to misunderstanding or misinterpreting. The examples should also be used as acceptance test cases, in which case they help detect design or coding errors (although unit tests should identify most of these first).

Conclusion

I encourage you to choose one or more of these practices to adopt in your current development work. There will be extra effort initially to understand and become comfortable with a given practice, but this will decline over time as you achieve mastery of it.

• Ask people to help.
• Look at what people do.
• Learn from the facts you gather.
• Try it yourself.

IDEO designs an incredibly wide variety of products – corporate websites, hand-held electronics, clothing, business services, furniture, and more. With such diversity, it is no surprise that not all of the cards appear applicable to software development. I found that going through the cards and thinking about how they relate to I.T. to be interesting. I ended up classifying the relevant cards into three groups:

• Requirements: These cards provide ideas on how to gather or analyze requirements. This was the largest group by far – over one third of the total number of cards, and they spanned the four categories above. If you work in I.T., it may seem that calling tips for requirements design approaches is inappropriate. In software development there is often a divide between requirements and design. You do need to understand the client's needs and determine a solution that meets those needs, but an explicit separation in role between these activities can often hurt the final product. IDEO takes a holistic approach: determining requirements and understand the user is part of the design process and not a precursor to it.
• User Interface Design: About one-sixth of the cards presented ideas related to user interface design. These mostly fell into the Try category, with a few in Learn and Ask. Prototyping and testing of various sorts were reoccurring themes in over half of these cards.
• Software Design: Only a few cards seemed relevant to application architecture and software design. I initially found this surprising since I was expecting more. After further reflection, I realized that the commonly held understanding of design in the context of software development is very technical and narrowly focused. This 'technical' design activity (for lack of a better term) is necessary but not sufficient for creating a great piece of software. Other activities within the course of a development project that we in I.T. do not call design – activities such as requirements gathering, analysis, and usability testing – are all part of IDEO's holistic view of design.

In order to provide a concrete example of what the cards are like, I list in the table below a few of the cards I found particularly interesting. Each card explains not only a design method (the how), but also the reason for using it (the why). (Each card also briefly describes an IDEO project that used this method, but I do not list that below.)

Looking at these design methods and the many others listed in the set of IDEO cards makes me appreciate all that goes into a well-designed product, and inspires me to think more carefully about how I think about and approach design.

People I have dealt with professionally who are not software developers occasionally make comments that imply that writing enterprise business software is a fairly straight-forward task, easily handled by the average developer. I have challenged these statements by pointing out that reality disagrees. The majority of software development projects are overrun or canceled. Software that does make it into production often has serious operational failures, performance issues, or fails to meet users' requirements, and much of it is poorly designed, making it difficult to correct these issues and implement enhancements. These problems cannot be blamed solely upon weak developers. Across the industry, the typical software development team will be staffed by average developers (by definition), which means that the problems I just described affecting a majority of projects result from the efforts of average developers.

In defense of software developers, the leading causes of project failure or overrun are actually in the areas of project management (i.e. overly aggressive schedules) and requirements gathering. Problems in these areas cause much more serious impacts, and this is perhaps why project managers and business analysts sometimes feel that writing the code is the easy part. But it is not. Even assuming sufficient time, clear requirements, and a decent solution architecture (which are big assumptions), producing working, high-quality code is hard.

Why is this? I'll first look at meeting the functional requirements. For a typical enterprise business application many of the functional requirements are admittedly simple, and can be summarized as creating data entry screens for a particular set of data. Many of the enterprise business applications I have been involved with, however, also have more complex functional requirements such as complex business rules, batch processing, web services, or data conversion. These more challenging requirements are the ones most likely to cause problems for the development team.

Software must also meet non-functional requirements in areas such as performance, usability, and maintainability, and I believe that most enterprise software does poorly in these areas. It is simple to explain why: everyone is focused on meeting the functional requirements - clients, testers, and project managers as well as developers - so the non-functional requirements are neglected. Even when someone - most often the architect - considers the non-functional requirements, it is difficult to address them all at the same time. Typically multiple iterations are necessary. For example, a developer first writes code that meets the functional requirements, then tries out the user interface and make improvements to address usability issues, then tests and addresses performance issues, and finally cleans up the code to ensure the design is maintainable - easy to understand and modify. These multiple iterations take time, skill and dedication, and if a developer is lacking in one of these areas, the non-functional requirements will likely be neglected.

Combine the challenges in both the functional and non-functional requirements, and the result is that the typical software application will fall short in at least one area. Add in other common challenges like a compressed schedule and unclear or changing requirements, and the chance of success plummets even lower. In other words, producing good software is hard.