The role that I want to share today is the architect as a scout in the military sense. A definition from Google is "a person sent out ahead of a main force so as to gather information about the enemy's position, strength, or movements." I like the military metaphor because software development is a team activity - the architect as scout is serving the rest of the team with the objective of helping keep the team moving ahead at a steady pace. This means being aware of what lies ahead, mapping the path forward, avoiding or eliminating obstacles, and being an initial target for hostile fire.

Awareness of what lies ahead

Architects need to be aware of what lies ahead in terms of both technical and business changes. The more accurate their picture of the future, the better their designs and decisions can be.

Awareness of the technical domain includes staying current on new libraries, frameworks, and platforms (e.g. mobile), learning about new software development practices (e.g. DevOps), and knowing what new infrastructure might become available to the team (e.g. middleware upgrades, cloud).

Awareness of the business domain includes having an idea of what major new functionality the business is thinking of introducing (e.g. need for multidimensional data analysis), and also having an idea of what business events or changes might significantly impact the system (e.g. significant variations in business usage of the system from how it was originally modeled, resulting in a significant change in the load and/or data volumes).

As an example, I recently recommended planning to do some refactoring to eliminate some duplicate code in the generation logic for two reports. Since there was a lot of this repeated code and it was clearly identified and isolated making it an easy, low risk refactoring, I though this was an obvious decision. Unfortunately, after making the recommendation, I learned that the business was expecting to eliminate one of the two reports in the near future. This completely changed the value proposition of the refactoring, actually making it potentially harmful in that it might increase the work required to rip out the one report entirely down the road. As a result, I reversed my recommendation.

Mapping the path forward

Architects determine how to use new technologies and how to solution new requirements to map out a clear path forward for the rest of the team. By tackling the risky or unknown items early, architects help the team maintain consistent progress.

Architects can use spikes, experiments, and pilots to try out and validate new technologies, solutions, and practices. Sometimes architects cannot do more than a basic evaluation in complete isolation. One approach I have used is to then do a pilot with a single team while closely monitoring and providing guidance, and then evaluate the suitability for rolling the change out more broadly within the organization. In such a case the entire team doing the pilot becomes a scouting force for the entire organization, led by the architect.

Solutioning of major new requirements can involve build-versus-buy decisions, determining major new components or infrastructure required, and architectural-level design. The common objective is to clarify what the team will need to do and allow for advance planning for activities needing more lead time (e.g. provisioning of new infrastructure, installing new software, training on new technologies).

For agile teams mapping the path forward typically means working an iteration earlier, ahead of the majority of the team, to ensure that business requirements are ready for the team to work on in the coming iteration and to ensure that there are minimal surprises with any new technologies or solutions. Architects can also be a source of technical requirements that are part of the product backlog. These activities help ensure that the team maintains a consistent, predictable velocity in delivering business value. Some agile teams might not have someone named explicitly in the role of architect, but all teams at least occasionally need someone operating as a scout to chart the waters ahead.

Dealing with obstacles

Architects need to identify and then avoid or eliminate obstacles that will potentially prevent successful delivery of the software or impede the team's progress.

Sometimes these obstacles are tricky technical issues that the team is having trouble resolving. As an example, one time I became involved in solving a performance problem with a batch process that took ever-increasing amounts of time to process individual records as the batch process ran. The problem was hard to find because it was not due to any particular piece of application code, but to how Hibernate (the underlying object-relational mapping library) was being used. To fix the problem I had to adjust the overall design of the batch process to reduce caching within Hibernate (follow the link to read more about the technical details).

Obstacles sometimes arise when the business raises requirements that are overly complex or extremely challenging to implement, especially in terms of meeting non-functional requirements such as performance or scalability. Architects sometimes need to push back on these requirements, or at least raise the alarm regarding the higher risk to ensure that sufficient time and resources are allocated to dealing with it.

Obstacles are not just with the software being developed or its requirements - they can also arise with respect to the project or external stakeholder expectations. For example, if management imposes an unrealistic delivery schedule on the team then architects are responsible for raising the alarm and helping determine ways to mitigate the risk.

Target for hostile fire

Architects serve as lightning rods for the team to initially bear the brunt of external problems or threats, especially those technical in nature. In this role architects act as a buffer to avoid having the team distracted or demotivated by these types of issues.

While the term 'hostile fire' is a metaphor, sometimes there are people in the organization who really are hostile. This may be due to political fallout from unpopular decisions, or stakeholder unhappiness with the software (e.g. due to poor performance or crippling defects). When the issues are technical the architect is the first contact - hearing the complaints and taking responsibility for what the team has produced. This is seldom pleasant, but is one of the responsibilities that comes with the role.

Sometimes the hostile fire really is a metaphor for tools or technologies that are too bleeding-edge or otherwise do not work as advertised. The architect then must confront the pain points and determine whether it is manageable or whether the tool must be scrapped and an alternative selected. Hostility may arise as a result of these issues or the resulting decisions, either from the team itself if they discovered these issues using a product proposed by the architect, or from parties within the organization outside of the team that are promoting or mandating the use of the problematic tool.

One example of this is years ago when new version control software was being introduced to replace CVS. I had not been involved with the initial pilot, but had heard disturbing things about the tool despite the decision to move ahead with its adoption. So I volunteered my team to be one of the first adopters and basically did my own evaluation. Shortly after transitioning to the new tool, I discovered a major design flaw that made it unusable. Efforts to work with the vendor to get the problem rectified failed, so I presented the situation to management and the decision to abandon the tool was made.

In conclusion, if you are an architect or functioning as a scout for your team ensure you are aware of what lies ahead, map out the path forward, deal with obstacles, and be the first contact for hostile fire.

Visualizing package dependencies can be useful for a number of reasons:

• Reverse engineering the architecture of an application when documentation is limited, out-of-date, or non-existent.
• Automatically generating diagrams, saving the time required to draw them manually.
• To understand impacts when doing large-scale refactorings or changing implementations of components.
• To determine how to extract functionality for reuse.
• To support good architecture / design as espoused by Robert C. Martin in his book Agile Software Development, Principles, Patterns, and Practices, such as by eliminating cyclical dependencies or unwanted dependencies.

Here is an example package dependency diagram generated by Graphviz:

The source code provided below that was used to generate this example diagram is available from my Java Examples project on GitHub or as a packaged bundle on my Software page. The code can easily be reused: change the directory of compiled code scanned by JDepend ("dist/classes"), and change the root package used to limit the scope of the diagram ("com.basilv.examples.packagediagram").

package com.basilv.examples.packagediagram;
import java.io.*;
import java.util.*;
import jdepend.framework.*;

public class PackageDiagramCreatorApp {

  public static void main(String[] args) {
    createPackageDependencyDiagram();
    System.exit(0);
  }

  public static void createPackageDependencyDiagram() {
    Collection packages = analyzePackages();
    StringBuilder builder = generateGraph(packages);
    generateImage("packages", builder.toString());
  }

  @SuppressWarnings("unchecked")
  private static Collection analyzePackages() {
    JDepend jdepend = new JDepend();
    try {
      jdepend.addDirectory("dist/classes");
    } catch (IOException e) {
      throw new RuntimeException("Error adding directory for JDepend to analyze.", e);
    }
    Collection packages = jdepend.analyze();
    return packages;
  }

  private static StringBuilder generateGraph(
    Collection packages) {
    StringBuilder builder = new StringBuilder();
    builder.append("digraph packages {").append("\n");
    builder.append("node [shape=box];").append("\n");
    builder.append("rankdir=BT;").append("\n");
    Set drawnDependencies = new HashSet();
    for (JavaPackage javaPackage : packages) {
      String packageNodeName = getGraphVizNodeForPackage(javaPackage);
      if (packageNodeName == null) {
        continue;
      }
      builder.append(packageNodeName).append("\n");

      @SuppressWarnings("unchecked")
      Collection dependencies = javaPackage.getEfferents();

      for (JavaPackage dependency : dependencies) {
        String dependencyNodeName = getGraphVizNodeForPackage(dependency);
        if (dependencyNodeName == null
          || packageNodeName.equals(dependencyNodeName)) {
          continue;
        }
        String dependencyKey = packageNodeName + "->"
          + dependencyNodeName;
        if (drawnDependencies.contains(dependencyKey)) {
          continue;
        }
        builder.append(packageNodeName).append(" -> ").append(
          dependencyNodeName).append(" [weight=4]").append("\n");
        drawnDependencies.add(dependencyKey);
      }
    }
    builder.append("}\n");
    return builder;
  }

  private static String getGraphVizNodeForPackage(
    JavaPackage javaPackage) {

    String rootPackage = "com.basilv.examples.packagediagram";
    String packageName = javaPackage.getName();
    if (!packageName.startsWith(rootPackage)) {
      return null;
    }

    return packageName.replace(".", "_");
  }

  private static void generateImage(String fileName,
    String graphVizDotFormattedGraph) {
    try {
      File graphFile = createFileWithContents(fileName
        + ".txt", graphVizDotFormattedGraph);

      // This requires the Graphviz software to be installed -
      // see http://graphviz.org/
      String imageFileLocation = fileName + ".png";
      Runtime.getRuntime().exec(
        "dot -v -Tpng " + graphFile.getName() + " -o "
          + imageFileLocation);

      System.out.println("Image file available at "
        + new File(imageFileLocation).getAbsolutePath());
    } catch (IOException e) {
      throw new RuntimeException("Error generating image " + fileName, e);
    }
  }

  private static File createFileWithContents(
    String fileName, String graphVizDotFormattedGraph)
    throws IOException {
    File graphFile = new File(fileName);
    FileWriter writer = new FileWriter(graphFile, false);
    try {
      writer.append(graphVizDotFormattedGraph);
    } finally {
      writer.close();
    }
    return graphFile;
  }

}

What is SOA?

There are many definitions of SOA. (See for example Wikipedia's definitions.) I consider SOA to be an enterprise architecture strategy where business logic is organized into services, and business processes are realized through the invocation of these services. I consider a service in the context of SOA to be a related set of automated business activities accessible through a well-defined interface. Services are often expected to be accessible remotely over the network, usually as web services using the HTTP for the transport protocol, SOAP for the message format, and compliant with one or more of the 'million' WS-* specifications. "SOA is more about good design and architecture than it is technology." (from Lawrence Wilkes). This is often confused, unfortunately, by all the vendor hype around particular products and specifications, especially web services.

A successful service oriented architecture needs to address three areas:

1. Services: the units of composition for a service oriented architecture.
2. Processes: the business processes using the services. Each business process is implemented by orchestrating together multiple services.
3. Governance: the policies governing the development and maintenance of services.

I am hesitant to call SOA an architecture pattern. A pattern is generally defined as a reoccurring solution to a reoccurring problem. SOA qualifies as a reoccurring solution, but it is not clear whether it addresses a single reoccurring problem. The more hype-driven literature assumes that SOA is a given without any justification. In other cases, SOA is presented as addressing one or more of a collection of enterprise issues without any discussion of the context around these issues and the consequences of using SOA. I think SOA could be expressed as an architectural pattern, and that doing so would be beneficial in reducing the hype and the sometimes contentious discussion surrounding SOA. I feel that looking at the motivation, applicability, and consequences of SOA would be particularly helpful, and in the remainder of this article I will touch on these points.

Does SOA represent something new? I have heard it said that SOA is merely a set of "best practices" for enterprise I.T. - a collection of what has worked before. Certainly some details of SOA like the enterprise service bus (ESB) seem like a rehash of the message bus in enterprise application integration (EAI). The inclusion of business processes in SOA, especially executable via the business process execution language (BPEL), comes straight from the notion of automated work-flow engines. From the middle-ware vendors' perspectives SOA means a new 'stack' of products to sell. I suspect this has been a major driving force behind the SOA hype. As existing middle-ware such as Java EE servers becomes commoditized and open-sourced, it is rational for vendors to create new, higher-profit markets where this has not yet happened. My perspective is that SOA is at least an incremental refinement of what has come before: it does pull formerly disparate concepts together into a unified whole. Its explicit inclusion of process and governance seems unique, although I remain uncertain how those aspects will work out in practice. Whether or not SOA is something new or just the product of vendor hype, the question "does SOA work?" remains. Below I look at some of the benefits and weaknesses of SOA.

Benefits of SOA

SOA promises many benefits, but how many of these can be achieved in practice? Below I list the reported benefits of using SOA and evaluate each one.

• Integration of legacy or stovepipe applications: Legacy or stovepipe applications, even those implemented in variant technologies, can be wrapped in a service and orchestrated with other, perhaps newer, services to achieve a business process. This mixing of old and new allows existing I.T. assets to be reused rather than needing to be rewritten, which is expected to result in cost savings. I feel this is a legitimate benefit, especially when such an application is not under the control of your organizational unit and cannot therefore be modified directly.
• Bridging the divide between I.T. and the business: SOA explicitly considers business processes and thus links business goals and business use cases with the I.T. architecture through services. While I think SOA goes much farther in this direction than other architectures, I suspect that many of the normal issues that arise between I.T. and the business will remain. The people involved will still come from primarily from one side or the other and bring their existing viewpoints and biases which will inherently lead to conflicts.
• Loose coupling: SOA encourages loose coupling through encapsulating functionality behind course-grained services with well defined interfaces. This allows the implementations of services to change without impacting users of the service, since other services and business processes will only invoke the service through its interface. This is similar conceptually to the use of dependency injection, where the code using an external resource knows nothing about the concrete implementation since it is injected by the container. While I think this is a significant benefit, bad design can still ruin it.
• Reuse: Reuse is one of the more highly touted benefits of SOA. The theory is that the services can be reused to implement new business processes. Since each service operation represents a business activity, its reuse represents a significant savings in development costs. This is the benefit I am most skeptical of. It seems far too similar to the promises made when object-oriented programming was first becoming mainstream. While OO has many benefits, the level of reuse that does occur is far below what was initially hyped. There are a few hopeful signs, however, that SOA may actually enable a reasonable level of reuse. One of the key requirements to enable reuse is the proper management support, which SOA explicitly addresses through governance. Services are course-grained, so even a large corporation will have a limited number of them (less than 500 is a ballpark I have heard). This makes it easier to review the catalog of services and determine possibilities for reuse. And since the service operations have business meaning, both I.T. management and the business can better understand and commit to their reuse, unlike in OO where individual classes or even libraries are at too technical a level to be truly comprehensible to such people. A significant obstacle to service reuse, however, is that it requires the organization's business processes to become more standardized in how various business activities are performed, since the same services to perform these activities will be invoked by these different processes. In a typical large organization with independently-minded business units, perhaps with varying requirements, this standardization will likely be difficult to achieve without senior management involvement.

Weaknesses of SOA

There are a number of drawbacks and challenges to successfully using SOA, of which I highlight what I feel are the most significant:

• Distributed Systems Problems: Making a system distributed is problematic. Martin Fowler's first law of distributed object design is "Don't do it" for this reason. The issues you need to deal with include the following:
  • Reliability: If any required service in your business process becomes unavailable then your entire business process cannot be executed. Even having systems with different scheduled maintenance windows can cause difficulties. One way to address this is to use asynchronous messaging as much as possible and make your messaging infrastructure high availability.
  • Performance: The performance of remote calls is much, much worse than local (in-process) calls. This is exasperated when using Web Services with the verbosity and extra parsing that XML and the various WS-* specs add. This is why service operations should be as course-grained as possible to minimize the number of remote invocations.
  • Transactions: For a typical application, database transactions are used to ensure data integrity. When multiple, remote services are being invoked this becomes much more difficult. One option is to use a distributed transaction, which requires that both the service messaging infrastructure (i.e. Web Services) supports distributed transactions, and that the underlying service implementations can enlist in these transactions. When wrapping services around legacy or off-the-shelf applications, this may not be possible. Even when distributed transactions can be made to work, they introduce yet another performance hit. Another option is to forgot about distributed transactions and deal with the consequences: this usually involves more careful design and the use of compensating transactions in the case of failures. It sounds dangerous to those such as myself used to working with transactions, but there are cases where transaction-limited systems have been made.
  • Security: The current security context - the identity of the user and the actions it is authorized to perform – often needs to be propagated to each service. Propagation is actually not that difficult, especially with the introduction of specifications like WS-Security which automatically include security context in the SOAP header. A harder problem is managing user identity and access privileges across separate, perhaps legacy, systems. Even if you introduce an enterprise-wide security service for all your applications to use, you still face the problem of dealing with off-the-shelf or legacy applications that do not use it.
• Changing Services: Once services are defined, created and in operational use, changing them becomes difficult. One problem is versioning: what happens when the interface to a service needs to change? Especially troubling are interface changes that are non-backwards compatible. Frameworks in languages such as Java with explicit interfaces have already had to deal with this problem. One solution is to change all the consumers of the service at the same time the service changes: this requires significant organizational coordination and is frequently unrealistic. Another solution is to create a version 2.0 of the service, and leave the old service running but deprecated to allow time for consumers to migrate to the new service. This requires that both versions of the service are operational, which increases maintenance costs and is not always possible. Service implementations can also be changed without touching the interface. While a purported benefit of SOA, there is the risk that an implementation change actually subtly changes the semantics of one or more operations and breaks one or more of the consumers of the service.
• Crossing Organizational Boundaries: For all the benefits of SOA to be realized, a number of organizational boundaries need to be bridged. This typically requires a change in organization culture, which is very difficult to achieve. Integrating services in separate, traditionally stovepipe, organizational units requires them to coordinate more closely, both during the initial development project and on an ongoing basis for maintenance. Effective management and reuse of services requires more effective communication between development teams and architecture within I.T, and potentially requires better alignment of separate organizational business processes with enterprise-wide processes. I've already mentioned bridging the divide between the business and I.T. as one of the reported benefits of SOA that is likely challenging to achieve in practice.

Other Viewpoints

As part of my research into SOA, I came across a number of sites with varying views and feedback:

• What is SOA? (wanderingbarque.com)
• What is SOA? - SOA and Web Services Explained (www.indiawebdevelopers.com)
• SOA Manifesto (dotnetaddict.dotnetdevelopersjournal.com)
• Can I Be of Service? (webservices.sys-con.com)
• Tactics, not strategy, drive SOA adoption (www.looselycoupled.com)
• Early adopters: SOA worth the effort (www.networkworld.com)

What is your take on SOA? Please leave a comment and share your experiences.

Why is deployability important?

Maintaining the proper operation of the production systems is a fundamental I.T. goal. The end users do not care how well the software works in the development or test environments. All the testing and other quality assurance activities in the world will not help if the software is not promoted properly into production and fails as a result. Ease of deployment is also important. If promoting changes is a cumbersome and labor-intensive task, people will be tempted to take shortcuts (i.e. skipping the test environment), and it will take longer to get changes into production.

How can you architect for deployability?

My first guideline is to minimize differences between environments. The more similar the environments, the simpler and more reliable it is to deploy the software. Each difference between environments is something that may trip you up when deploying if you have failed to properly address it. Environmental differences can also affect the accuracy of testing: what worked in one environment may not work in a different one. Unfortunately, it is not possible to eliminate all differences without eliminating all environments but one. Common unavoidable differences between environments include different servers, different hardware configurations, different databases, different URLs, and different security settings. When encountering an environmental difference, I ask the question "Is there a good reason for this difference to exist?". If not, then it should be eliminated.

One example from my own experience is a system deployed to Unix servers which had a standard directory structure except for the root directory that differed across environments. In my investigation I discovered that the reason for the difference was that the test server contained two separate test environments, each of which therefore required a different root directory. I considered this a valid reason given the existing hardware at the time. When the opportunity arose, I suggested that an additional test server be added and the second test environment moved to it to allow the paths to be standardized.

To handle unavoidable environmental differences, I recommend encapsulating them within the software to isolate these differences from the rest of the system. The specific mechanism will vary depending on the technology being used and the particular difference being addressed. I discuss some of these mechanisms in my article Designing for Deployability. The goal you should aim for is to be able to deploy the software using a simple, automated process. For deploying code, this typically means an automated script that takes the output of the build process and copies it to the appropriate location (usually on another server) for the environment you are deploying to.

In an earlier article on Deploying Application Changes, I distinguished between full deployments where the entire application or component is deployed as a unit, and partial deployments in which only the delta - the pieces that have changed - is promoted. As I discuss in that article, there are advantages and disadvantages to either approach. Within a single application, I prefer the full deployment approach. When multiple applications are involved, especially across multiple support or business units, I prefer to have the option of partial deployments. If separate applications are too-tightly coupled, then changes in one will require changes in the other, which then requires that both applications be deployed together. So high coupling between applications reduces the ease of deployability of each individual application. Clearly defined interfaces help minimize coupling, and if the changing application maintains backwards-compatibility of its interface, the other application will then not need to change, or can choose to change on its own time line.

Like most non-functional requirements, deployability is often neglected or overlooked during application development. Inexperienced or rushed developers can create systems that work great in development, but have many problems in the test or production environments due to issues with deployment. The absence of a defined and preferably automated deploy process or problems in testing due to environment differences are warning signs that more attention to deployability is needed.

The first session was Lessons for Risk Management Taken from the Titanic, presented by Mark Kozak-Holland. I really enjoyed his use of the story of the Titanic as a case study to relate to modern IT projects. Mark talked about a variety of project management issues and not just risk management as the title suggested. Some of his most noteworthy points were:

• When do projects really finish? Most software development projects are considered complete when the software is deployed into production, if not earlier. Mark argued, and I agree, that the solution should be operational in production for a period of time for the project to be considered done. For the project to be considered a true success, the key return-on-investment metrics that drove the project should be measured to demonstrate that the original purpose of the project was achieved.
• There are three different phases of a project when failures can happen: during the development, during implementation, and during operation. The later the phase, the more costly the failure. The key to successful operation is to ensure that non-functional requirements such as maintainability, performance, security, manageability, etc. get the same focus during development and testing as the functional requirements. Most executives and business representatives understand the functional requirements, but seldom understand the non-functional requirements, so there is a natural tendency for them to be neglected - either cut from the project scope, or not sufficiently tested. Mark pointed out that it is often much easier to add on functional requirements than non-functional requirements, and emphasized that project managers and architects should push back when the sponsors or business makes decisions that would compromise the solution.

The second session was The Role of the Enterprise Architect, presented by Jason Uppal. I have had conflicting opinions regarding the work and value of architects, so it was refreshing to hear Jason present a very pragmatic and reasonable viewpoint of enterprise architects. Some of the points that I liked were:

• Enterprise architecture starts with a vision - is this project / solution the right thing to do? Jason had a great example of an architecture review he did of a overdue project. He determined that the original business goal of the project had been to improve the functionality and performance of a set of business reports. But the actual project being delivered was to migrate the mainframe application to a more modern web-based application.
• Jason had an interesting rule of thumb regarding the problem statement provided by a client: shorter is better. A one sentence problem statement is ideal, while a multi-page document spells doom for the project. Why? The longer the problem statement, the more the client is specifying the solution they think they want, rather than specifying the actual problem. The more the client specifies the solution, the less they make use of the expertise of the architect in determining the solution.
• Jason made an interesting point regarding project cancellation: it should be considered operational learning rather than a failure. One of the responsibilities of the architect is to determine whether the project makes sense. If it doesn't, it should be killed - the earlier the better.
• Architects should never be the cause of project delays. Each architectural iteration should be one to two weeks in length. Each iteration elaborates on what was produced before and leaves fewer unanswered questions.
• Architects should not have assumptions in their work. Instead, any assumptions should be treated and identified as risks. This allows for risk management strategies - i.e. risk mitigation - to plan for the risk if it occurs.

Actually, my last sentence is not true. Customers do want software that works, and since simpler software is more reliable, they benefit from fighting complexity. Unfortunately, the costs of complexity are largely hidden from the sight of the customer, so they seldom realize the cost involved in asking for more features. They just get upset when the software stops working or works poorly, and they do not appreciate their contribution to the problem. IT operational staff also benefit from fighting complexity to keep systems as reliable as possible, since they need to keep them running. But they seldom have much if any influence upon the procurement or development of these systems.

Lately I have been struggling with improving the reliability of a particular system. As the team has identified and tried to resolve various issues, I have come to see that the high complexity of the system overshadows our efforts. Why does complexity so strongly affect reliability? I like using a mechanical analogy: the more moving parts in a device, the higher the probability that one of them will fail within a fixed time, thus lowering the overall reliability of the device. In an IT system, the failure points are different. The actual physical devices - the hardware - is ironically simpler to manage since it is easy to improve through redundancy. It is the software that is the problem. The greater the complexity of the software, the higher the likelihood of defects - not just within the application code itself, but also in the overall software stack that is used. For an enterprise business application, this typically includes third party libraries, application server, web server, database server, and operating system, and can include additional services such as email, scheduling or messaging. A defect anywhere in the stack can cause the application to fail.

The problem with software reliability goes beyond just defects. In an enterprise setting, applications experience a wide variety of changes, each of which represents an opportunity for failure. Each of these changes is in essence a "moving part", even though the actual code for the application has not changed. The most typical change is enhancements to the application, which can introduce new defects in both the new and existing functionality. Other examples of changes include upgrades to application servers, web servers, database servers, operating systems, or hardware, configuration changes to systems such as email, network addresses, or scheduling, or security changes such as password expiration. The more complex the system, the more of these changes it experiences, which increases the chance of failure.

The relationship between complexity and reliability can be modeled statistically. I will represent an IT system as a collection of pieces (P) that each has a chance of failure (F), expressed as a probability of failing within one year. I think of each piece as abstractly representing something that can failure - the equivalent of that moving part in a mechanical device. This correlates with the complexity of the system. While it is hard to determine even approximate values for these measures in a real system, just using abstract concepts and figures can provide an appreciation for the relationship between the two values. The probability of the system having no failures in one year is (1-F)^P. Using baseline values of 100 pieces and a 0.01 probability of failure for each piece in the year (1%), the chance of no failures in a year is only 37%. This means the chance of having one or more failures is 63%. What happens as the complexity increases?

The reliability of the system falls quickly as the number of pieces is increased. In order to maintain the same reliability when the complexity doubles, the reliability of each piece must double.

In practice, however, more complex systems are harder to understand and change, thus reducing the reliability of each change that is made. Once a system does fail, greater complexity means that it is often harder to diagnose and fix the problem. This makes the downtime longer. Complexity therefore also leads to more serious failures.

Complexity and reliability are closely connected. If you have no plan to manage the complexity of a system, then you may be unpleasantly surprised by what happens to its reliability. Since our goal as professionals is to provide software that works, thinking about complexity and reliability is a necessity.