Questioning Testing

A fictional dialogue is presented below to explore and question the traditional role of the I.T. quality assurance department. The characters are:

  • Bess: Manager of a line of business.
  • Asa: Manager of the I.T. quality assurance (QA) department.

The scenario is that Bess has asked I.T. to provide some software to monitor and drive improvements to business operations. Since no suitable commercial product exists, I.T. has assembled a development team to build it. During the project inception, Bess is concerned by the proposed project schedule and in particular is confused by the need to allocate a large block of time prior to the first release for quality assurance testing. So Bess contacts Asa to try to understand why this is necessary.

Bess: Why does QA need all this time at the end of the release for testing?

Asa: QA needs to ensure that the software being built will work properly once it is delivered to your unit to use.

Bess: That sounds great, but I thought the development team was doing testing?

Asa: Oh yes, of course they are.

Bess: So why do you need to do more testing?

Asa: There are two reasons. The first is that we work to understand the needs of your business unit and collaborate with your people to test the software to ensure it meets your users' needs. We call this user acceptance testing. The second reason is that we cannot always trust the development team to do a good job. Sometimes they are rushed, or sometimes I.T. hires contractors or gets new developers that don't work out. So QA is the final quality gate prior to the software being delivered to you.

Bess: That is a helpful explanation of what your group does, but I'm still confused as to why this needs to be done at the end and why it takes so much time.

Asa: That's easy, we need to wait until the development team completes the software before testing it. And we are very thorough in our testing - that takes a lot of time, you know.

Bess: Now I'm more confused. You said the development team was testing - are they doing this before your group tests, or at the same time?

Asa: Oh no, we can't have them testing at the same time as us. They need to be finished their testing before we test.

Bess: But why? It seems like this is the key reason the overall schedule is so long. Why can't your group start testing at the same time as the development team?

Asa: Well, strictly speaking the developers are testing all along, as they write the code, before the system is usable. Once enough of the software is finished, the development team tests that the whole system works. They call this system testing. During this entire process, they are finding and fixing defects, and the system can be unstable. If my people were to be testing at the same time, they would have their testing disrupted by this and spend extra effort retesting. That wastes their time. So it is far more efficient for us if we wait until they finish their testing.

Bess: But what are your testers doing while they wait? Don't they have to be involved from early on in the project to understand my business needs for that - what did you call it - user acceptance testing?

Asa: Yes, my testers are involved from basically the start of the project to understand the requirements and the proposed solution. However, they have a lot to do, you know. They have to create their test plans and prepare their test scripts and test data. They produce a lot of documentation - they don't just do testing.

Bess: So essentially your testers spend all that time preparing to test?

Asa: Yes, that's right.

Bess: So how is it that the development team can not only prepare to do their own testing, but actually perform their testing, deal with defects, and finish retesting all before your testers have finished preparing? It sounds to me like waiting to test till the end has made your testers less efficient, not more efficient.

Asa: Well, ....

Bess: Hold on, I'm not finished. I actually have a bigger concern. You are worried about the efficiency of the testers in your unit, but they are only part of a larger process of getting software to me, which in my world is just the start of a series of steps towards improving business operations. Shouldn't you be worried about optimizing the entire process, not just your unit?

Asa: I'm not sure I understand what you are getting at.

Bess: Haven't you heard of lean thinking? Optimize the whole value stream? Eliminate waste?

Asa: ...

Bess: I guess not. Let me put this in regular business terms. I expect this software will help my unit save millions of dollars each month. Yes, each month! And you are telling me that because it is supposedly more efficient for a few of your testers, you want to push out the schedule by a few months? Unless your testers assigned to this project cost millions each month, I don't need to do a return on investment analysis to tell me that extending the schedule is a bad business decision.

Asa: But..., but what about user acceptance testing?

Bess: Ah, thanks for bringing that up. I wanted to discuss that more. How does it make sense to wait until the end of the release - after the development team has finished testing - to ensure that the needs of my business line and end users are met? If they aren't, doesn't this mean that the development team would need to make changes that would push the schedule out even more?

Asa: Ummm...

Bess: In fact, when I talked earlier to the development team they said that every few weeks they are planning to provide regular demos of the software they are building to help ensure it will meet our needs.

Asa: Well, that isn't the same as user acceptance testing.

Bess: Good point. The development team mentioned something about providing us a sandbox for us to play with the system, which I didn't really understand at the time. But maybe we can use it to do some of this user testing throughout the project.

Asa: But your users are not skilled testers.

Bess: No, but if I understood you earlier, you provide the skilled testers who will collaborate with my users to help with the testing.

Asa: Yes, but that early in the project? What about defects? System instability?

Bess: (Sigh) If the development team is producing poor quality work, it makes sense to find out as soon as possible. After all, quality has to be built into the process. Sorry, that's another lean reference.

Asa: But there needs to be a formal user acceptance test of the final product - that's part of our quality gate process.

Bess: Well, if we have been involved throughout the project - seeing demos, playing with the system, and doing testing - then I don't see why the final user testing should take much time at all. And we should be able to do it at the same time as the development team does their final system testing.

Asa: But our process calls for system testing of the final product to be complete before formal user acceptance testing starts!

Bess: It sounds to me like you should change your process. Let me explain my perspective. You want to avoid the risk of defects found during final system testing from impacting the effort spent in formal user acceptance testing. But this risk mitigation introduces a schedule impact to your customer - that's me, by the way - which is an issue: a guaranteed impact to how soon my unit can begin to use the software. And this risk is only realized if defects are found that are serious enough to warrant fixing at this late stage. Given all the testing done throughout the project, the likelihood of this risk should be quite low. So in summary you want to exchange a low severity risk for a higher impact issue? From my point of view that is definitely the wrong business decision. I'd rather have final user acceptance testing start at the same time as final system testing, and accept the low-impact risk of extra testing effort due to defect fixes in exchange for a shorter overall schedule. In fact, I'll pay for any extra time your testers have to spend retesting during the final user acceptance testing. There, problem solved.

Asa: But... pay for my staff... umm... can you do that?

Bess: Indirectly, don't I help do that already, through the revenue brought in by my line of business? Doesn't I.T. exist to help support the business in achieving their objectives, instead of putting up artificial obstacles?

Asa: ...

Bess: I'm going to wrap up this conversation. You have been most helpful in explaining the situation from your perspective, so thank you for your time. I will be following up with the CIO to ensure that my concerns are addressed. And I should talk again with the development team to see if there are ways for them to speed up as well, without sacrificing quality.

If you find this article helpful, please make a donation.

«