«    »

The Trouble with Traceability

In software development traceability is the linkage of requirements to the software and/or development artifacts like design or test cases. The underlying objective is to ensure that everything the customer or user requires has been correctly delivered. I have no quibbles with this goal, but in practice the applications of traceability I have seen leave me feeling troubled. I often wonder whether there are sufficient benefits achieved to justify the efforts spent on traceability, especially formal traceability with defined artifacts.

Formal Traceability

The most frequent incarnation of formal traceability I see is the requirements traceability matrix (RTM). This is a two-dimensional table, often represented by a spreadsheet, in which the one dimension lists requirements and the other dimension lists the artifact being traced to - business objectives, test cases, or design documentation. Cells within the table are filled in with a mark when the corresponding artifact and requirement are related.

The example traceability matrix below lists requirements across as columns and test cases down as rows. Notice that requirement 2.2 has no corresponding related test case, which is considered a traceability gap that indicates that this requirement has not been tested.

Req 1.1 Req 1.2 Req 1.3 Req 2.1 Req 2.2
TC 1 X
TC 2 X X
TC 3 X
TC 4 X
TC 5 X
TC 6 X
TC 7 X X X X
TC 8 X X

Traceability Troubles

Below are listed the aspects of traceability that trouble me.

  • I have successfully developed and enhanced high quality software applications that have pleased clients and been used in production without needing formal traceability. So does it really provide value? Traceability seems like it addresses the problem of having unimplemented or untested requirements. This has rarely been a problem on the development projects I have been on. We are much more likely to struggle with ambiguous or unclear requirements, missed or new requirements, and incorrect logic in obscure special conditions.
  • The tracing of requirements to test cases assumes they can be identified and listed which in turn limits the testing approaches that can be used. The context-driven school of testing would challenge these limitations. Exploratory testing in particular does not fit the confines imposed by a formal requirements traceability matrix.
  • While the value of producing the initial RTM is unclear to me at best, I am even more dubious about the value of maintaining a RTM over time. The typical claims is that for for future requirement changes (e.g. enhancements) the RTM can be used to do impact analysis and determine what tests need to be executed and/or changed. Regarding impact analysis, the RTM might get you started, but the analysis must always consider code-level dependencies not reflected in the RTM (through, for example, reuse of common functionality), and must consider other requirements and functionality that are not directly related (and thus not reflected in the RTM) but make assumptions that are no longer true under the proposed change. The idea that the RTM can be used to determine what tests to executes makes several poor assumptions. First, that the tests are manual and not automated. If you have an automated test suite, you can just rerun the whole suite to identify failing tests - you never need to worry about running a subset. Second, that reusing manual tests is a good idea. If a test is worth re-executing often, it is frequently worth automating. For those tests that are manual, much of the benefit of having human testers is lost if they simply re-run the same scripts over and over, since there is a low probability of such re-executions failing and testers are less likely to find defects when they do occur. A better approach is to let the tester come up with their own variations each time to try to creatively break the system.
  • Tracing requirements to design is conceptually flawed. It at best very indirectly addresses the objective of traceability - ensuring the customer gets the functionality they require. Doing this mapping is in reality quite hard because the mapping from design to code (and from requirements to design) is not a strict one-to-one and is often a complex many-to-many, especially for holistic non-functional requirements like performance or security. Furthermore, the idea of tracing to design assumes that there are fine-grained pieces of design that can be identified and referenced. This presumes a certain approach to design and its documentation that is at odds with many modern methods of software development.
  • For small-scale software applications, the system is simple enough that producing a RTM is a reasonably small effort. The number of requirements and test cases is relatively small. But the chance of errors occurring that would be caught be traceability is likewise low because the system's complexity is low. For very large systems with hundreds or thousands of requirements and tests, it is far more conceivable that requirements may be forgotten to be implemented or tested. But while the benefits of formal traceability are higher, the efforts are much, much higher. For a RTM, the number of cells to complete essentially scales non-linearly with the number of requirements in the system.
  • I do not understand the obsession with testing in the context of traceability and especially RTMs. Reviews or inspections are much more effective at finding defects than testing. So why do I never hear of formal traceability from requirements to reviews?

Conclusion

Software engineering researchers and industry leaders have reached the same conclusions about traceability. Andrew Kannenberg and Dr. Hossein Saiedian in Why Software Requirements Traceability Remains a Challenge (PDF) are clearly big proponents of requirements traceability, but look at their conclusions:

  • "Unfortunately, manual traceability methods are not suitable for the needs of the software engineering industry."
  • "Currently existing COTS traceability tools are not adequate for the needs of the software engineering industry."

Robert Glass in his book Facts and Fallacies of Software Engineering points out the underlying cause of troubles implementing traceability: "when a requirement commonly links to 50 or more design requirements and each of those links to some very much larger number of coding elements and coding elements may be reused to satisfy more than one requirement, we get a burgeoning complexity problem that has resisted manual solution and even tended to thwart most automated solutions" (page 78)

I hope the above discussion has convinced you to tread cautiously when it comes to traceability. In a future article I will look at alternatives to formal traceability that achieve the same underlying objectives.

If you find this article helpful, please make a donation.

«    »