• Measuring Defect Potentials and Defect Removal (pdf)
• Software Assessments, Benchmarks, and Best Practices
• Estimating Software Costs: Bringing Realism to Estimating

The information I provide in the remainder of this article comes primarily from these references.

Estimating Software Size

The first step in Caper's approach is to estimate the size of the software. His preferred size metric is function points, which are a language/technology neutral evaluation of the business functionality based primarily on an assessment of program inputs, program outputs, and data storage. Another common metric is logical lines of code, often abbreviated as SLOC. KLOC represents 1000 lines of code.

Lines of code are a convenient metric in that they can be measured automatically with a tool, whereas function points required a trained function point counter. Nevertheless, function points are superiour in several key ways:

• Roughly fifty percent of defects are due to problems with requirements or design rather than coding, for which metrics in terms of lines of code do not make much sense. In particular, different implementations of the same functionality can have significant variances in the lines of code required by up to a factor of four. A longer implementation will have fewer requirement or design defects per KLOC and thus artificially seem to have higher quality, when in reality the code is simply bloated. (This is related to the issue of measuring developer productivity by lines of code produced.)
• The use of multiple languages complicates source code counts, and this is far more common than one may expect. Even a simple web application typically includes JavaScript, HTML, CSS, and perhaps SQL in addition to the primary language (e.g. Java). Function points are language-independent.
• Function points can be determined once the design is known - lines of code require waiting till coding be completed. This allows function points to be useful for planning and estimation purposes much earlier.

Since Caper prefers function points, the defect metrics he provides in his writings are typically expressed using this measure. While I understood the reasons, I found these metrics difficult to apply because I had no idea of what the function point counts were of the software I was working on. So I was pleased to discover that benchmarks such as this one exist to convert between function points and lines of code for various languages. One function point on average is roughly 50 logical lines of Java code. I use this conversion factor in the sections below.

Defect Potential

All software has the potential of having defects. Defect potential is a measurement of the expected number of defects in a particular piece of software. This is also called the injection rate - the number of defects being introduced throughout development. The primary factor determining the number of defects is the size of the application in function points. The maturity - experience, skill, and attention to quality - of the development team is another key factor in determining defect potential. The following table specifies how to calculate the expected number of defects given these two factors.

Some key defect prevention activities contributing to reduced defect potential are:

• Close customer collaboration during requirements / design (e.g. JAD sessions)
• Prototyping
• Feedback / learning from design and code reviews

The above metrics assume a linear relationship between defects and size, but as a system gets larger there are typicaly more interactions between pieces and more complexity, and thus a greater likelihood of defects than a linear increase would suggest. For very large systems, a more accurate metric is as follows: number of defects = function points raised to an exponent. For average organizations, use 1.25 as the exponent. (Good organizations can lower this to 1.15, while poor-performing organizations have this elevated to 1.35.)

Defects can be categorized by origin - the type of activity that produced the defect. The table below shows this breakdown for average organizations.

Defect Removal

Defect removal is the identification and elimination of defects after they are introduced. The cumulative defect removal rate or defect removal efficiency of a development project is calculated as the number of defects eliminated prior to the release to production divided by the total number of defects found after 90 days of production use.

The following table shows how the defect removal rate varies with the maturity level of the team, just like defect potential, and shows the expected number of post-release defects based on the defect potential and removal metrics.

Removal efficiency varies for defects of different origins, as the following table shows using statistics for average organizations.

Quality control procedures such as testing and reviews (inspections) vary in their effectiveness at removing defects as illustrated in the following table.

Peak defect removal rates for a given activity are typically obtained only through the use of skilled, experienced staff who take a rigourous, disciplined approach to performing the activity. As an example consider unit testing. As it is normally performed it has a 25% defect removal rate. But an experienced developer following test-driven development will achieve nearly 100% code coverage and typically write better tests, leading to a higher 50% defect removal rate.

Predicting Defect Levels

The overall or cumulative defect removal rate for a develoment effort can be calculated by aggregating together the individual defect removal rates of the quality control procedures used by the team as follows: Cumulative removal rate = 1 - the product across all procedures of (1 - individual removal rate per procedure).

For example, if a team uses only unit testing (25% removal), functional testing (30%), and regression testing (20%), then the cumulative rate = 1 - (1-0.25) * (1-0.30) * (1-0.20) = 0.58 or 58%.

The overall number of defects in production (or UAT) can be calculated using the defect potential of the team to determine the expected number of defects introduced, and using the cumulative defect removal rate to determine the number of defects remaining. Approximately 25% of defects will be high severity.

For example, an average organization injecting 1 defect per 10 lines of code for a 25 KLOC application will end up introducing a total of 2500 defects. Given a cumulative defect removal rate of 95%, this means that 2375 defects will be found, leaving 125 defects remaining of which 31 can be expected to be high severity.

Achieving the full definition of done for a feature or release (often called being done-done) is surprisingly difficult. You may have heard the anecdotal rule of thumb that it takes 80% of the time to achieve 80% complete on a feature, and another 80% time-wise to achieve the remaining 20%. This is not due to using a strict definition of done. Anyone who has worked on an undisciplined waterfall project that has 'finished' coding the functionality and is now in test knows that sinking feeling of having defect after defect reported with no end in sight – and hence no predictability.

Using an imprecise or overly-simplistic definition of done will definitely impede predictability because you have only a vague idea of what the current state of your software is, and hence have high uncertainty regarding your progress to date.

Using a specific definition of done combined with the lean principle of minimizing work-in-progress greatly improves your knowledge of the current state of your software and hence your progress. Most of the uncertainty has been driven out by forcing work to done-done. This is not enough, however, to improve the predictability of your budget or schedule estimates for your outstanding work. You need to ensure your planned tasks and estimates include everything required to get to done-done.

On my last project, we tried just tracking coding tasks in the sprint plan. The remaining effort to get to done-done was treated as part of the general overhead that was used to calculate our velocity. This provided poor predictability in terms of both budget and schedule, especially as a feature neared code-complete. The estimate on the remaining coding tasks approached zero, but the time required for reviews, testing, and fixes remained roughly constant, thus exceeding what could be accounted for by velocity.

We then started to track most checklist items from our definition of done as individual tasks for each feature in addition to the regular coding tasks provided by the developer(s). This worked better, but we still suffered predictability problems as the feature went through review and test. Defects discovered had to be fixed, impacting the schedule and budget for the feature.

The next tweak we made was to allocate time up front for these fixes by adding an entry titled "Anticipated Issues" to each feature that provided an allocation of effort (budget, from which schedule was derived) that could be used to address defects, review corrections, poorly estimated tasks, and discovered work. The amount of effort allocated to this item was estimated per feature based on the size and complexity of the feature and the experience and quality level of the developers working on the feature.

Our most recent adjustment was to the front-end of feature development. Over several sprints, we noticed a trend that when developers started a new feature, especially one that was significantly new, they often had a day or even two where they did not make the expected progress on their tasks. We eventually determined that developers consistently had to spend time familiarizing themselves with the feature, especially learning the requirements and getting clarifications regarding how the feature was to be implemented, and that they virtually never allocated time in their tasks / estimates for this. So we added a new task to each feature named "Feature Clarification" to make this explicit.

By planning for all the components of our definition of done, we have achieved much better predictability. While our variance from our estimates per feature remained fairly high, the average variance across features was much closer to the original estimate, rather than being consistently over. As progress is made on a feature, we have much better estimates and better visibility into the outstanding work.

I have already alluded to an essential concept: creating an estimate is attempting to predict the future. The basis of all estimation techniques is to extrapolate the future based on work done in the past, so the higher the likelihood that the future matches the past, the more accurate the estimate. How well the future matches the past is determined by a number of factors:

• How well can you predict what work must be done in the future? In other words, how clearly do you understand the scope and requirements for the activity you need to estimate? I do not feel that software development is much different in this regard from other fields of endeavor: whenever you ask a trade for an estimate, they first ask questions to determine the size of the job to be done. For construction projects, there are blueprints that define the scope of what needs to be built. Even a well-defined requirements specification, unfortunately, is not enough, even ignoring the likely scenario of changing requirements. Within software development, requirements do not actually define the work that needs to be done, just how the system is to function. It is the architecture and design of the system that determines the work necessary to construct it. While a high level design (i.e. architecture) can be done initially, this is still insufficient for an accurate estimate. A detailed or low-level design is necessary to clearly identify the work to be done, and will typically need revisions throughout construction (assuming it is done up front).
• How similar is the work to be estimated to work done in the past? The more change from before, the more difficult it will be to estimate. This is why research or discovery types of activities are essentially impossible to estimate: if it has never been done before, there is no real basis for an estimate. Activities involving lots of thinking and creativity are similar. Significant portions of software development projects fall into this category, including requirements analysis, design, and defect fixing. I am a proponent of the viewpoint that coding is itself often a creative design activity: writing code is the final step in specifying the design of the system. (For an excellent argument in favor of this viewpoint, check out the article What is Software Design?.) Software development also experiences a rapid rate of change in terms of tools and technology: the work that needs to be done might be similar to that done five years ago, but if it is using new technology then an effective estimate is much more difficult.
• Who will be performing the work? It is difficult enough to estimate how long a software development activity will take for yourself, let along for someone else whose skill level and experience you have no idea of. This is particularly significant given that software engineering research has identified that developer productivity can vary by as much as an order of magnitude - 10 times difference between the best and worst, and 2 times difference between the top and bottom halves (See the book Peopleware: Productive Projects and Teams starting at page 44 for more details.)

Making an accurate estimate is difficult. I define estimate accuracy as the percentage difference between the estimated and actual value. Lower numbers are better. For example, an estimate of 4 months that actually took 6 months had an accuracy (difference) of 50%. An estimate of 4 months that took 4.5 months was much more accurate at 12.5% difference. This implies that the actual accuracy of an estimate can only be evaluated when the activity is completed. Estimates can (and should) include a predicted accuracy, such as 4 months +/- 50%, but note that this predicated accuracy is itself an estimate. One of my favorite flippant responses when asked for an estimate is to say it will take less than two years. I haven't been wrong yet :) Unfortunately, like predictions of the future, estimates of such low accuracy hold no value.

Management typically insists on accurate estimates in order to have firm schedules, cost projections, and resource utilization plans. In the worst case I have seen there was a service level agreement requiring an estimate accuracy of plus or minus five percent. I consider requiring such accurate estimates to be unreasonable for the simple reason that the more accurate the estimate must be, the more effort must be expended to create that estimate. And the only effective way to significantly improve the accuracy of an estimate is to proceed with the activities or project to be estimated. Steve McConnell does an excellent job explaining this in his book Rapid Development, starting at page 165. Essentially, a project starts out with a fuzzy conception of what needs to be done, so any estimate done in this phase is likewise fuzzy. As the project proceeds and the final product becomes clearer, the estimate can become more accurate. Research quoted by Steve indicates estimate accuracies are around 400% at initial product concept, 50% after requirements, and fall to 10% after detailed design is completed. To achieve that 5% estimate accuracy would likely require over half of the coding to be completed.

Lack of awareness about the difficulty of estimating leads to unrealistic expectations and demands. This in turn saddles projects with unrealistic budgets or schedules from day one, setting them up for failure. I believe the I.T. industry would benefit greatly from an increased understanding of estimation and its inherent difficulty. Realizing that an estimate is nothing more than a prediction of the future whose accuracy depends on how clearly our understanding of the future matches how it unfolds is the first step.

These two types of time are related - the larger the effort involved the longer it will take, all else being equal. But they measure different quantities, even though both are expressed in units of time. A common pitfall is to conflate the two. Developers often estimate their tasks in effort time, while project managers and customers are more concerned with elapsed time. The following story highlights this problem.

Project manager Mark comes by developer Dave's cubicle Monday morning to ask him how long it will take to complete the foo-bar feature. Dave figures there are about five days worth of work, so he tells Mark it will take a week. Mark tells Dave to start working on it and walks away thinking it will be done by next Monday. As the week passes, Dave makes good progress on the feature, but is nowhere near done by next Monday. When Mark comes by to verify that the foo-bar feature is finished, he is disappointed to hear that Dave isn't done. Dave defends himself by explaining that he spent time on other activities that left him with less time than he expected to work on the foo-bar feature. Tuesday he had a design session to go to. Wednesday was the regular team meeting, which took longer than normal. Thursday Tim the tester found a defect in the previous feature Dave had implemented, and it took Dave a few hours to fix the problem. Dave concludes by saying that as a result of these other tasks, he hasn't yet put in his full five days of work on the feature, and figures he will still achieve his original estimate.

What went wrong? Dave's estimate was in effort time, but Mark assumed it was in elapsed time. The root cause was a communication failure. Whenever I'm asked for an estimate, I always clarify whether elapsed time or effort time is wanted.

What if you have created your estimate in effort time, but have been asked when you'll be finished? How do you convert between effort and elapsed time? I'll start with effort time and present some formulas for converting to elapsed time. Effort time is assumed to be in units of hours of effort by one person, totaled for the task or project for which elapsed time is desired. A simplistic conversion formula is:

Elapsed time in calendar days = (total effort) * 1.4 / (number of hours worked per day * number of people)

This formula first converts the effort in hours to days of effort for the whole team, then converts that to elapsed time using the factor of 1.4, which is derived from 7 days in a week / 5 working days in a week. The final value can be used to determine a project completion date by adding it to the project's start date.

Here's an example. A project is estimated to have 160 hours worth of tasks. There are 2 developers each working 8 hour days. The elapsed time = 160 * 1.4 / (8 * 2) = 14 calendar days or 2 weeks.

Unfortunately, this simplistic formula will underestimate the actual elapsed time because it fails to account for several factors. The first factor is that the amount of time a person spends working on a project is always less than the amount of time spent at work. I call this the productivity level: the ratio or percentage of time spent working on the project versus the total time spent at work. The ideal is 100%. The practical maximum is about 95%, which means that out of a 40 hour work week, 2 hours a week are spent on overhead or administrative activities. Even for a developer dedicated to a single project, there are emails to process, meetings to attend, time reporting to attend to, communication with other developers, and other similar tasks that prevent them from obtaining 100% productivity. People assigned multiple responsibilities fare much worse. Overhead activities increase roughly linearly with ones responsibilities. For example, if you are involved with two projects instead of one then you likely have twice the number of emails to process and separate meetings for each project to attend. There is more task switching, and an increased likelihood of interruption since there are more people you are dealing with who may have need to communicate with you. In my experience, the productivity level in this case can fall to 75% or lower.

Another factor affecting elapsed time is that employees don't always work five days a week, especially for longer projects of several months duration or more. The simplistic formula's factor of 1.4 only accounts for weekends. There are also statutory holidays, sick days, vacation days, and training days to consider. We can calculate a new factor to include this other time off as follows. Start with 365 days in the year - 52*2 (weekends) - 10 (stat holidays) - 15 (3 weeks vacation) - 5 (sick days) - 1 (training) = 230 days worked in the year. The ratio 365 / 230 = 1.6.

Revising the simplistic formula to account for these additional factors results in a more accurate formula for calculating elapsed time from effort time:

Elapsed time in calendar days = (total effort) * 1.6 / (number of hours worked per day * number of people * average team productivity level)

Using our earlier example (160 hours of effort, 2 developers, 8 hour work day) assuming a 75% productivity level gives the following. Elapsed time = 160 * 1.6 / (2 * 8 * 0.75) = 21 days or roughly 4 weeks. This is twice as long as the result produced by the simplistic formula.

This revised formula is still only an approximation and could be further refined. For example, large teams have increased management and communication overhead that should be explicitly factored in. For multi-year projects turnover of project members becomes a significant factor. In my experience, however, the above formula is sufficient for modestly-sized software development projects - up to ten people working for no more than one year.