«    »

Balancing Order and Chaos with Process

Whether it is software development projects or I.T. operations, many larger organizations seem enamored with process as the solution to their problems. The default reaction to negative outcomes or variances between people in performing activities is to add more process. Process provides an ordered structure for keeping such chaos at bay. This reflects an underlying value system that believes that chaos is always bad and order is always good.

The fundamental flaw of this all-too-common reasoning is the implicit assumption that there are no drawbacks to order and process. But there are drawbacks, as I will show shortly. And if order has drawbacks, then this must mean that there are corresponding benefits to chaos. Chaos can be good! Yes, I know this is a provocative statement given the strong negative connotation of the word. But chaos is simply defined as a state of disorder. The second law of thermodynamics states that entropy (disorder) increases over time in a closed system. So chaos is more natural than order. And anything natural must be good, right? :)

On a more serious note, the second law of thermodynamics points towards a key drawback of order and process: it requires organizational energy to be achieved. This law applies to human systems just as much as physical systems. You cannot expect to obtain a higher-ordered state without expending more energy. For example, consider the typical example of adding a new process for employees to follow. If you expend minimal energy and just update a process document, this will have virtually no impact. You need education or training on the new process and oversight (e.g. followup, audits, etc.) to ensure it is correctly understood and continues to be followed. This all takes effort and time. Even with self-organizing teams you cannot avoid the second law: rather than management expending effort to drive and enforce process, it will be expended by each team.

The third law of thermodynamics states that the entropy (disorder) of matter approaches zero as the temperature approaches zero. This makes sense since temperature is essentially the amount of random motion of atoms and molecules. When you have no random motion (a temperature of zero) you have no disorder. But what state is such matter in? It is a solid - a very frozen solid, even for normally fluid substances like air. And such matter is very brittle: sharp impacts will cause it to shatter (e.g. use a hammer to hit a banana dipped in liquid carbon dioxide). Using this as an analogy for human systems points to a second key drawback of order: it reduces the ability to change. The more energy expended in establishing and maintaining a given process, the harder and slower it is to change. Rapid or sudden change can be impossible to deal with.

I experienced this first-hand when dealing with a ISO-compliant quality system with an overly-burdensome process for tracking improvements. It was a common occurrence at the regular quality meeting to review improvements that the quality manager would bemoan the almost complete lack of improvements. So at one meeting I mentioned an improvement that I was in the middle of implementing. Well, the quality manager became quite excited - someone was doing an improvement! So this person emailed me immediately after the meeting and asked me to fill out a two-page template to describe the improvement. I had not seen this template before nor it did appear obvious / easy to fill out, so I resisted. But it became obvious to me why so few improvements were being formally identified: the process was too heavy-weight, and people did not perceive a benefit for the effort required. A few years later, a tool was introduced for tracking improvements that made it extremely trivial to record an improvement, and suddenly we had many more improvements being entered and tracked.

So the key disadvantages of more process are that it takes more effort to achieve and reduces the ability to change. Thus, when considering adding more process, a cost-benefit analysis needs to be done. The problem is that many people are short-sighted thinkers. They have an immediate problem, and see that the additional process will address their immediate issue. They don't look beyond and see that the effort involved in ensuring compliance to the process and the reduced agility in dealing with change. They don't consider that future changes to other processes will have to factor in impacts on this new process. So the bias is always towards more process, rather than seeking an appropriate balance between order and chaos.

Strangely, my experience suggests that it is the management roles within an organization that have the most difficulty seeking an appropriate balance. I suspect that this is partly because the essential role of management is to establish structure and order, and partly because they seldom are forced to perform the processes they insist on and thus don't experience on a daily basis the extra effort required to comply.

Some people might point to automation as the solution: automate a process, and you don't need to expend effort performing it or ensuring it is done consistently. Computers are great at that, unlike humans. I agree that automation is often great: I love automated test suites, automated builds, and automated deployments. But it takes effort to create the automation, and it is inflexible - typically much harder to change than a manual human process. In other words, the drawbacks of higher order are still in force: you cannot cheat the laws of thermodynamics.

Seeking this point of balance between order and chaos is not easy. I led a recent project that had very aggressive, inflexible timelines and a very high rate of discovery in building a solution to meet the business needs. These tough constraints forced me to discard all except the most critical process in order to maximize the speed of delivery while being as responsive as possible to business changes. However, I still needed to ensure that a high quality product was delivered, which is very difficult to do with short timelines and frequent changes. I needed just barely enough light-weight process to keep quality high without overly impacting forward progress. It felt like I was sprinting along the edge of a cliff where the slightest slip could spell disaster. Yet in the end balance was maintained and my team delivered a quality system within the required timeline.

The point of balance will vary between teams and projects, based on their unique characteristics and based on the cost-benefit analysis regarding the amount and types of process required. This is another point of conflict in larger organizations where there is often a push for consistency of process that typically results in extra process being applied to teams that do not need it and may even be harmed by it. This point of balance can also change over time, requiring organizations to periodically re-evaluate the need for less or more process.

In the end, it is all about seeking and maintaining that ideal point of balance between order and chaos through having the right amount and right kind of process.

If you find this article helpful, please make a donation.

«    »