Based on my experience, I thought I had a good appreciation for what can go wrong. That changed recently when I experienced a day filled with too many problems and errors to be believed. The day started off innocently enough until a member of the production support team came by to inform us that he had accidentally terminated the database connection of one of our batch jobs due to transposing two numbers in the identifier. This by fluke matched our job instead of the one he wanted. Okay, no problem, we simply need to confirm that nothing was corrupted and restart the process. We checked our email for the notification email that is sent when a batch job abnormally terminates in order to verify which job had been affected. No such email was found. A little puzzled, we checked the server and confirmed that the process was no longer running. But another batch process was executing, and we identified it as a subsequent batch job dependent on the first. Subsequent jobs only run if the predecessors execute successfully, so we had a sinking feeling as we started checking the log files. Sure enough, due to a complete lack of error handling, the first job had reported a successful execution despite the database connection failure, which had caused the second job to start. That explained the lack of a notification email. The second job depended on the processing results of the first job, so the output of the second job was suspect and likely wrong. We had to kill the second job. If the first job had just failed, we could have restarted it without a problem, but now we had to investigate how to undo the effects of both jobs and manually restart the first.

Well, that didn't seem too bad, until I had time to think for a second. That is when I realized that our batch jobs are always scheduled at night or the weekends, and never during business hours. What was one doing running during the day? That prompted another investigation, which revealed that the job does normally runs on weekends. But the previous weekend there were problems with predecessors to this job that caused it to be delayed until one of the nights during the week. So why didn't it run at night? We were surprised to discover that it had – it just didn't finish. Due to performance issues, the job had run for over eight hours, extending into the day, before it was killed by mistake.

By late afternoon of that day we had multiple investigations underway trying to track down the various root causes of the problems we had identified. My mind reached the saturation point sometime in the afternoon, so I cannot remember the details concerning what was found. I suspect there were other problems unearthed that I have since forgotten. Nor where we able to get everything fixed that same day. That combination of problems, coming together on that one day, kept a surprising number of people busy for days sorting out the mess.

Why is deployability important?

Maintaining the proper operation of the production systems is a fundamental I.T. goal. The end users do not care how well the software works in the development or test environments. All the testing and other quality assurance activities in the world will not help if the software is not promoted properly into production and fails as a result. Ease of deployment is also important. If promoting changes is a cumbersome and labor-intensive task, people will be tempted to take shortcuts (i.e. skipping the test environment), and it will take longer to get changes into production.

How can you architect for deployability?

My first guideline is to minimize differences between environments. The more similar the environments, the simpler and more reliable it is to deploy the software. Each difference between environments is something that may trip you up when deploying if you have failed to properly address it. Environmental differences can also affect the accuracy of testing: what worked in one environment may not work in a different one. Unfortunately, it is not possible to eliminate all differences without eliminating all environments but one. Common unavoidable differences between environments include different servers, different hardware configurations, different databases, different URLs, and different security settings. When encountering an environmental difference, I ask the question "Is there a good reason for this difference to exist?". If not, then it should be eliminated.

One example from my own experience is a system deployed to Unix servers which had a standard directory structure except for the root directory that differed across environments. In my investigation I discovered that the reason for the difference was that the test server contained two separate test environments, each of which therefore required a different root directory. I considered this a valid reason given the existing hardware at the time. When the opportunity arose, I suggested that an additional test server be added and the second test environment moved to it to allow the paths to be standardized.

To handle unavoidable environmental differences, I recommend encapsulating them within the software to isolate these differences from the rest of the system. The specific mechanism will vary depending on the technology being used and the particular difference being addressed. I discuss some of these mechanisms in my article Designing for Deployability. The goal you should aim for is to be able to deploy the software using a simple, automated process. For deploying code, this typically means an automated script that takes the output of the build process and copies it to the appropriate location (usually on another server) for the environment you are deploying to.

In an earlier article on Deploying Application Changes, I distinguished between full deployments where the entire application or component is deployed as a unit, and partial deployments in which only the delta - the pieces that have changed - is promoted. As I discuss in that article, there are advantages and disadvantages to either approach. Within a single application, I prefer the full deployment approach. When multiple applications are involved, especially across multiple support or business units, I prefer to have the option of partial deployments. If separate applications are too-tightly coupled, then changes in one will require changes in the other, which then requires that both applications be deployed together. So high coupling between applications reduces the ease of deployability of each individual application. Clearly defined interfaces help minimize coupling, and if the changing application maintains backwards-compatibility of its interface, the other application will then not need to change, or can choose to change on its own time line.

Like most non-functional requirements, deployability is often neglected or overlooked during application development. Inexperienced or rushed developers can create systems that work great in development, but have many problems in the test or production environments due to issues with deployment. The absence of a defined and preferably automated deploy process or problems in testing due to environment differences are warning signs that more attention to deployability is needed.

This application deployment process must specify the set of application artifacts (i.e. executable code, database objects, files) that is deployed for a particular release - what I call the unit of deployment. There are two basic approaches to defining a unit of deployment.

The first approach is a full deployment: packaging together the entire application as a single unit. As a Java developer, this seems like a natural approach: the compiled Java class files along with other application files can be bundled into a single JAR, WAR or EAR file. A typical enterprise business application, however, consists of more than just the code. For the database in particular, deploying all the tables and views for the entire application when doing a release is not practical. Therefore, the second approach to defining a unit of deployment is a partial deployment: only include those application artifacts that have changed. The remainder of the artifacts are left unmodified in the environment being deployed to. For the database, this means that only those tables or views that have been modified need to be included in the deployment.

These two approaches each have advantages and disadvantages. The following criteria are useful for determining which approach should be used in a particular situation:

• How confidently can you track the application changes? If it is hard or error-prone to track changes to artifacts, then a full deployment is safer. With a partial deployment, you need to know that all relevant changes made in the development environment are being promoted to the test environment, and then from the test to the production environment. Missing one or more artifacts in the deployment to production can cause defects or even a complete failure of the application.
• How much manual effort is required to do the deployment? Manual effort increases the likelihood of a failed deployment due to human error. A full deployment is easy to automate and should require little if any manual effort to maintain, since you always just grab the entire application. A partial deployment can be automated for a single change, but it is hard to have a reusable automatic deployment mechanism since the set of artifacts that change will vary from release to release. Such a mechanism needs to be able to determine automatically what has changed and deploy just those pieces. A good example of this is the automatic update mechanisms found in operating systems and applications - they only download a patch containing the changes rather than an entire copy of the updated application.
• How much time is required to do the deployment? The longer it takes to deploy a single artifact, the more important this criteria, and the more valuable the partial deployment approach becomes. Database changes are the best example: to drop and recreate every table in the database for a change would require a full backup and restore of the data, which will often take prohibitively longer than simply updating the few tables that actually changed. Online application updates are limited by network bandwidth, so they also benefit from a partial deployment.

My preference is for full deployments. Unless circumstances require a partial deployment, I feel that the default choice should be to deploy everything. Unfortunately, this is seldom possible in a complex enterprise environment. Many enterprise applications consist of a collection of components or services such as web application, database, batch processing, scheduling, reporting, and web services. Due to organizational or logistical reasons, it is seldom possible to do a full deployment of all of these components in a single release. At the application level, therefore, a partial deployment is often done. It is at the component level where it is easier to achieve full deployments of the individual components.

1. Initial investigation. My goal is to reproduce the reported problem in the application in my development environment. This may require obtaining more information from the user(s). If I can't reproduce it, I try to reason about possible causes to help me figure out what conditions are necessary to reproduce it. In hard cases, I may need to add extra error-checking or error-handling code to try to narrow down what is going on when the problem happens again.
2. Classify the problem. Many problems are defects: a flaw in the code or design that prevents the application from working as designed. Sometimes the user may be having troubles with the user interface - I classify this as a usability defect. Or the existing functionality may be too limited and the user is requesting something new - this I classify as an enhancement request. The distinction between defect, usability defect and enhancement is quite blurry. Often I will treat a usability defect like any other defect that needs to be fixed, but if such a defect requires significant user interface changes, then I classify it as an enhancement. If I determine that the reported problem is really an enhancement request, then I am finished with my defect fixing process (enhancements are typically handled through a different process).
3. Analyze the cause. My goal is to write an automated unit test that proves the existence of the defect by failing - i.e. I'd expect the test to pass if the defect didn't exist. I may need to do some analysis to determine what section of code is failing, and may need to iterate on this a few times if the defect is especially hard to pin down. For certain problems, including most usability defects, it is too difficult to write an effective test so I rely on manual testing and debugging.
4. Fix the defect. Now that I know the cause of the defect, I can fix it. This is where having a test for the defect really pays off, since I can run the tests after making my fix and verify that the problem is corrected by having the tests pass.
5. Learn from the defect. The defect may be fixed, but I'm not done. Each defect is a learning opportunity; it represents a failure in something I've done that I can correct and improve. So I ask myself questions like the following. Why or how was this defect introduced into the code? Was it an error in design, a copy-and-paste error, a failure to consider special conditions like a null return value, improperly understood library, etc.? Why didn't the unit tests catch this problem? Why didn't reviews or manual testing catch this problem? Would similar defects exist elsewhere in the code, due to the same or a related failure that caused this problem? What can I do to prevent this type of defect from happening again?
6. Act on my learning. Based on what I've learned from this defect, I take action to improve myself, the application, and the team's processes for the future. Possible actions include writing more unit tests for the problematic section of code, refactoring a difficult-to-use API to eliminate the problem that occurred, informing other developers of why this defect happened, and improving error handling to make it easier to reproduce similar defects in the future.

What are the specific challenges facing developers maintaining software?

1. They probably did not originally develop the software, and may not even have access to the people who did. This is due to several factors. In some organizations the software maintenance group is separate from the software development group. Even when the same organizational group develops and maintains the software, it is common for the development team to be much larger than the number of people allocated to maintain the software, so the majority of the development team moves on to other projects. Another factor is that the lifespan of the software (five or more years) is usually longer than the lifespan of employees with the organization (average of three years).
2. They may need to make emergency fixes or release defect fixes quickly, while concurrently working on new features.
3. They must ensure that the software operates properly in its production environment, while developing and testing defect fixes and new features.

So how to address these challenges?

1. Make code readability a primary goal when writing code (after correctness). Rapid Development by Steve McConnell, page 255-256, relates an interesting experiment in which five development teams were tasked with writing the same piece of software. Each team was given a different objective (memory use, output readability, program readability, minimum statements, minimum programming time). Four of the five teams came in first place with respect to their objective, but no team came in first place in more than two of the objectives. The lesson: clearly define your objective, and make sure it is the right objective.

   Why is readability so importance for maintenance? Code you write is likely obvious to yourself, but try waiting a few months without looking at it, and then see what you think. And when it is another developer who has to look at the code, without the ability to ask you what you did, then readability becomes even more important. Since developers spend more time reading code than writing it, it makes sense to optimize the most frequently used path.

2. Use automated testing. Automated suites of tests using tools such as JUnit have been widely popularized in the last seven years, yet I still encounter developers not using this practice. In a discussion last year on automated unit testing, a senior developer stated that he did not see the need for such testing - he had no problems getting his code to work. While I doubt the validity of this reason - I find automated unit testing to be quite beneficial in catching defects in my code - more important are the developers who inherit this individual's code and must make changes. Automated testing is one of the best practices for reducing the risk of introducing new defects when changing the code. Another benefit of automated unit testing is that the unit tests serve as a form of documentation to new developers that shows how the code being tested is expected to be used.
3. Use version control software to its full potential. Most projects I have been involved with have used version control software, yet few have used it to the fullest extent possible. One example of this is the use of branching. It is a common practice in open source software development teams to use branching to manage changes to the currently released version of the software while implementing new features in the new version - exactly the situation faced in maintenance. Yet in commercial development, I have encountered many developers who either are completely unfamiliar with the concept of branching, or who are uncomfortable with performing branching and related activities in the version control software they use. For such developers, I recommend reading the book Software Configuration Management Patterns by Stephen Berczuk - it does an excellent job covering the concepts and recommended practices concerning the use of version control software (which is a subset of configuration management). I also recommend experimenting with branching with your version control software to become comfortable with the process. Create a test project with some files in your version control repository. Make a branch of this project, then modify files in both the original version and the branch. Then try merging / copying changes between the branch and the original version.

   Another case where version control software is not used to its full potential concerns which files associated with the application are actually being stored using version control. All projects using version control store the code, but many projects do not store other files associated with the application, such as web pages, online help, build scripts, database scripts, and technical documentation. Instead I have seen such projects use what I call the Common Server Directory anti-pattern. A set of directories on a file server is made accessible to the team, and everyone adds and modifies files in this common area. Sometimes an effort is made to track versions, via directory or file names, and sometimes not. Since a file system is not designed with the features of version control software, the end result is not surprising. The list of problems I have seen or encountered personally spans the entire spectrum: one person's changes overwritten by another, file corruption (on Windows servers), accidental moving of a sub-directory, confusion over previous versions of a file - who made which changes at what time, among others. Unfortunately, the solution is simple - use version control software - but hard to implement. Many of the other files associated with an application are created and/or owned by non-developers: web designers, business analysts, database administrators, testers, or technical writers. It is quite uncommon for such individuals to be familiar with the concepts of version control, let alone be comfortable using it. And such individuals are often in different groups than the development team, making it difficult to mandate the use of version control. If you had success implementing version control in this kind of situation, I'd like to hear how you accomplished it.

4. Ensure that the software is well-designed. Some qualities of a good design in the context of software maintenance are: as simple as possible, easy to understand, easy to make changes, easy to test, and easy to operate (i.e. easy to deploy and monitor for problems). Unfortunately, if you inherit an existing piece of software to maintain, then you will have to deal with the existing design, which may not be all that you'd like it to be. Even with well-designed software, as you add new features, you may find that portions of the original design are no longer appropriate, or that additional capabilities have to be included in the design. This is when the technique of refactoring is useful. Refactoring is a disciplined approach to modifying software code without changing its external behavior. A great book on this topic is Refactoring: Improving the Design of Existing Code by Martin Fowler. The more new features you need to add to an existing application, the more important refactoring becomes.

These recommendations should ideally be carried out when the software is first being developed, but it is never too late to start implementing these ideas. For those who have some say in the deliverables produced by a development project, please make sure items such as the automated test suite are included and are verified as part of the delivery. In one architectural review I did of a development project, I found evidence that JUnit unit tests were used by the developers, but they were not supplied in the release. On another maintenance project, the JUnit tests were present, but the configuration information necessary to connect to the database when running the tests was not documented or preserved in version control. I never was able to determine the configuration used (the individual able to run the tests had moved on before I joined the project), and I had to redesign the database connection code as a result.

This article is long enough that I deliberately did not go into details on how to implement each of the recommendations. If you are interested in reading more about any of these topics, let me know.

1. Defect fixing. This is most common when software first goes into production use from development.
2. Changed or new business requirements. During development, projects can experience a 25% change in requirements (Rapid Development by Steve McConnell, page 331). Depending on the level of customer involvement during the development process and the handling of requirements changes during development, there can be even more requirements changes once the software goes into production. If a waterfall development methodology was used with the requirements fixed up front in the requirements phase, then the customer will likely have a backlog of changes to be made once the software is in actual use. If the customer had minimal involvement during the development of the software - i.e. if the first time they saw the software was in user acceptance testing, then they will likely have changes they'll want done once they actually sit down to use the software.
3. Changes to the software's execution context. By execution context, I refer to the hardware and other software required for a business application to run. This includes such things as the operating system, database, application/web server, programming libraries, and network. Changing the operating system version or hardware will often have minimal impact on a business application, but will require maintenance effort to test the application on the new configuration and migrate to it.

So if business software will spend much more time in maintenance mode than in development, and will need to be maintained / changed fairly frequently, then it seems obvious to me that it is important that such software be maintainable. I define maintainable software as software for which it is easy to develop and deploy changes with minimal risk and impact to existing users. This definition suggests another reason why maintainability is important: you don't want to make changes to software in production use that negatively affect users (i.e. by introducing new defects in previously working functionality). This is not an issue faced in development. A poorly-coded piece of software can often be hacked and 'duct-taped' together to pass user acceptance testing. But once such software is being used daily, it becomes more and more difficult (approaching impossible) to make required changes, typically because each defect fix unexpectedly introduces new defects.

Many of the software development books I have read spend little time talking about the challenges and importance of maintainable software, and typically assume a new (greenfield) software development project. I hope I have convinced you of the importance of creating maintainable software. In future articles I will discuss how to achieve this.