«    »

Drawbacks of Formal Audits

In heavily-regulated or bureaucratic environments formal audits are a common occurrence. Such audits typically consist of an auditor external to the team or organization who analyzes historical evidence of the work done to find non-conformities with respect to the documented process being audited.

To those with a bureaucratic mindset process and audits are the answer to every problem: if something goes wrong, add more process and then audit to ensure it is followed.

Audits serve a purpose, but they do have drawbacks. Over-reliance on audits can actually cause negative consequences to the organization which are often not taken into account by those pushing for more auditing. Audits should be designed to minimize these drawbacks, and the organization should introduce additional mitigations as necessary. In fact, I would go as far as saying that the use of audits should be kept to a minimum.

So what are these drawbacks? I have organized them into the following categories which are explored in detail in the remainder of this article.

  • Reduces Productivity
  • Harms Organizational Culture
  • Limits of Assessment

Reduces Productivity

I define productivity as the amount of value produced for stakeholders based on the effort expended. In other industries such as manufacturing with more concrete notions of value it is possible to effectively measure productivity. In I.T., however, this is difficult to do, and I have never heard of an I.T. audit that evaluates productivity, value produced, or even effort expended. Audits typically instead measure compliance to documented processes based on historical documentation regarding the work that was done. This disconnect inevitably leads to a reduction in productivity in the following ways:

Do work to pass audit rather than deliver value

Auditing documented evidence of following documented processes causes people to produce documentation, whether or not it provides value, in order to pass the audit. This is especially true when the work becomes about producing the documentation instead of producing value.

Assessing quality after the fact is wasteful

Two important concepts from lean thinking are that any time delay in a process is waste, and that quality should be built into the process to avoid the waste associated with undetected quality problems remaining in the work-in-progress. Performing an audit long after an activity has been performed violates both of these concepts. In some cases I have seen audits are performed months after the activity has been completed - for example, an audit of a software enhancement that has already been deployed to production. If problems are found, it is too late for that activity. And the follow-up to determine if the problem has been resolved typically does not happen until the next audit, which is an even longer delay.

Harms Organizational Culture

An organization in which formal audits are a common occurrence risks, in my opinion, serious harm to the organizational culture by shifting the culture towards a more bureaucratic, stagnant, and confrontational atmosphere. This can happen in a number of ways:

Shift of focus away from organizational objectives

People generally have difficulty focusing on more than one or two priorities at a time. When the focus is placed on passing audits, attention is diverted away from achieving the organization's objectives or purpose. This is similar to the problem faced by traditional project management - making scope, schedule, and budget the goal runs the risk of not actually meeting the business needs.

Compliance mindset rather than improvement mindset

Formal audits, especially when they are treated strictly by management, tends to lead to a fear-based compliance mindset. The unspoken message is "pass the audit, or else...". This is inimical to a mindset of continuous improvement, specifically the attitude of being willing to change the process when a better approach is found. In theory a process-centric mindset and a continuous improvement mindset can coexist - Toyota seems to be successful at this. But adding audits into the mix adds a rigidity or bureaucracy to processes that impedes continuous improvement. Worse is when people blindly follow process rather than question whether there is a better way.

Attitude of confrontation rather than collaboration

Formal audits almost always involve separate auditors external to the group being audited, and sometimes external to the organization. And these auditors are trained and motivated to find non-conformities - deviations from process. Auditors are not engaged to work with the group to understand what they have done and to help them do better. In fact, they are often instructed to remain aloof in order to remain impartial. This tends to lead to an attitude of confrontation rather than collaboration - an us-versus-them mentality - which means that any useful feedback that the auditors might have is at risk of being denied or ignored.

Discourages context-based expert judgement, creativity, and initiative

The Dreyfus model of skill acquisition explains the difference between beginners and experts. Having a clearly defined, step-by-step process to follow is essential for beginners, but has been demonstrated to actually reduce the performance of experts, who use their judgement based on the context at hand to determine what to do and how to do it.
Enforcing adherence to a common process also discourages creativity, initiative, and innovation. This can happen in a bureaucratic environment without audits, of course, but I believe that a regime of formal audits will exasperate the problem.

Limits of Assessment

Audits at their essence are about assessing a group with respect to some standard. This assessment has certain fundamental limitations due to the nature of audits, and over-reliance on audits risks developing an incomplete or inaccurate picture of reality. These limitations include:

Cannot assess intangibles

Audits only assess concrete items that are documented and completely ignore intangible aspects of work such as motivation, creativity, and initiative. (Some of these intangibles can be assessed by other 'formal' mechanisms - e.g. assessing motivation through surveys.) The corollary to the adage "you get what you measure" is that you risk not getting what you don't measure. This would not be a problem if the intangibles were not important. But software development is knowledge work, and the research is clear that people are by far the most significant factor determining productivity, far ahead of process. So relying on audits, which typically only look at process, ignores the more intangible, people factors.

Assessment is twice removed from objectives

Each group or organization has a purpose for existence, and objectives by which it tries to achieve that purpose. Audits do not assess this. Organizations put processes into place to support meeting these objectives. But audits, strictly speaking, do not assess this either. Audits assess the historical, documented evidence that the processes were followed. This means they are twice removed from the true objectives, which makes them an incredibly weak tool for helping an organization meet its objectives.

Audit results may be invalid

Audits suffer from four failure modes which can cause them to report invalid results:

  1. The group is actually following the process being audited, but is not correctly producing the evidence.
  2. The group deliberately deviates from the process in order to meet organizational objectives or in order to be more effective.
  3. The group produces the documentation used as evidence for following the process without properly following the process. They will pass the audit, but there is a hidden problem. Rubber-stamping, where a review or approval is given without any actual check or thought, is an example of this.
  4. The group properly follows the process and produces the evidence, but fails to achieve organizational objectives.

In conclusion, be very careful in how you use audits and watch out for the drawbacks. Maybe you should audit your use of audits :)

If you find this article helpful, please make a donation.

«    »