1. Thou shalt not break the build

This is the first commandment because breaking the build causes immediate problems for the rest of the team. Anyone updating from version control repository will have a non-functional local copy of the code and will be unable to continue with their work until the problem is fixed. A broken build also causes issues for others wanting to commit changes: they cannot verify that their changes do not break the code base. Doing an unrelated commit on top of a broken build risks introducing new breaks that compound the difficulty in getting the code base back to working order. (This exact problem happened quite recently on my project.)

The general principle is to not introduce problems in the code base that interfere with other development activities. Each team needs to have a specific, unambiguous definition. As a minimum standard I require code to always compile and pass 100% of the unit tests. While I prefer that automated integration and functional tests always pass, they are slower to run and depend on external systems, so pragmatically the occasional failure can happen.

An easy way to make your definition of a broken build explicit is to define an automated build process that performs this check. This build process can be executed by the developer prior to committing a change, and can also be configured to execute on a continuous integration server immediately after a change is committed to ensure the code base is not broken.

I like having a penalty in place for those that break the build. It can be a small monetary fine (that goes into the team pot for the next social event), receiving a mascot of shame, or being assigned an unpleasant task.

2. Thou shalt put everything in version control

All code and all related files must be in the version control repository. A new developer or a new instance of a continuous integration server should be able to check out a copy of the software and build a complete release from it.

Too often I have encountered code bases where some key files are not included. I have seen a number of projects not include third-party libraries in version control. This might be deemed acceptable if an enterprise library repository is used (using a tool such as Ivy), but in the cases I have seen the code base required a directory of library files to exist on each developer's local workstation without even a definition of the required versions of the libraries being used. Other common omissions include build scripts, configuration files, and documentation.

3. Thou shalt use the version control repository as the source of truth

The version control repository is the official source of truth for all versions of the software. Source code distributions may be created and distributed via shared network drives or via the web, but the source of truth remains the repository.

If multiple branches are being used within the repository then the purpose of each branch must be clearly defined. The typical approach is to use the trunk (mainline) for ongoing development work, and create branches for releases. So the trunk is the source of truth for the software overall, while each release branch is the source of truth for a particular release, and in particular the fixes and/or patches made for a release. Distributed version control systems such as Git, which in essence treat each local workspace as a separate branch, still need to follow these same rules to avoid complete chaos. (For example, running a continuous integration build requires at least one official branch to build from.)

The corollary to this commandment is that changes to source code not in the repository do not officially exist. Committing changes into the repository is therefore a critical part of doing development that is reflected in the next commandment.

4. Thou shalt commit thy changes daily

The principle behind this commandment is that the effort and difficulty involved in integrating separate changes increases non-linearly with the size of the changes, so it is best to integrate changes as often as possible. (This is an instance of the lean principle of flow which dictates minimizing large batch sizes and work-in-progress.) Requiring daily commits ensures that conflicts or duplication of effort are found quickly.

There are other benefits of daily commits. It forces developers to decompose their work into smaller, incremental pieces of work and ensure the quality of that work (since it cannot break the build as per commandment number one). Developers are more likely to use step-by-step refactoring rather than big-bang changes that are much riskier (and more likely to negatively impact the rest of the team). Daily commits provide a visible sign of progress per developer that can be used to assess when a developer is stuck or sidetracked (the commits tend to stop).

5. Thou shalt treat thy version control system as mission critical software

I have seen more than one company which did not treat their corporate version control system as mission-critical software. In one case the version control server was classified as a under-powered development server that could be bounced at any time or upgraded without prior testing. In another case the source code repository did not have robust backup and restore mechanisms in place.

If the version control server goes down it is safe to assume that regular development activities will start to grind to a halt within the day and any emergency fixes or scheduled releases will likely need to be delayed. It is a good idea to ensure that a version control server is available 24x7 with the ability to restore service within a half-day maximum. Why 24x7? Regular hours of development likely do not correspond to regular business hours (especially for emergency fixes), and scheduled builds typically run at night or the weekend.

I believe that producing good software is hard, and that we software developers need all the help we can get in developing software. I have put together a list of the top five practices that I find to be the most helpful in making it easier to produce good software:

• Automated Tests
• Version Control
• Refactoring
• Repeatable Build and Deploy
• Communication

These practices are generally applicable across the board: for new application development and ongoing application maintenance, for small teams and large teams, for custom enterprise software for a single customer and commercial shrink-wrap software, and for any language or technology. The specifics of how the practice is implemented will of course vary depending on the nature of the project.

Automated Tests

Automated unit tests provide a convenient way to test the software's functionality both during initial development and later as the software is maintained. This is especially essential during maintenance when the original developers are no longer around. Another benefit of automated tests is that they impose a design constraint upon the software that it be testable. This frequently helps improve the overall design of the software by reducing the amount of coupling between different components.

In order to be effective at finding defects, automated tests need to be written effectively. I have come across many examples of poorly written unit tests that are ineffective at finding defects. My article How to write good unit tests discusses this in detail.

Version Control

The use of version control software to manage changes to the software is indispensable for teams larger than one person. Some of the more important benefits of version control include helping coordinate concurrent changes to the code base, providing a history of changes, tracking the versions making up a release, and supporting parallel development on different versions of the software. I even use version control for my personal projects at home, where I am the sole developer. (Read about my experience using Subversion.)

Some of the problems I have observed when version control is not used are: each team member developing with a slightly different version of the software, resulting in chaos when it is time to deploy the software, and variations in the software installed in different environments, with no one able to explain for sure what the correct version is.

Refactoring

The design of a piece of software is a critical attribute for determining its quality. While good design does not imply good software, bad design generally does imply bad software. Refactoring is the key practice by which the quality of the design can be maintained and even incrementally improved as the software changes. Refactoring is a potentially risky activity because of the possibility of introducing defects. This risk is mitigated when the software has a comprehensive suite of automated tests.

Repeatable Build and Deploy

Every software team has some kind of method for building and deploying their software. I wanted to emphasize repeatability because that is the essential aspect of this practice. A build and deploy process that you can repeat consistently across team members, across environments, and across different versions of the software is critical to successfully shipping your software or deploying it into production. This practice only works effectively when version control is used to define the official version of the software being built and deployed.

A related aspect of this practice is keeping your active development mainline in good shape, which I define as having the code compile and pass all unit tests. This allows developers to reliably check out, build, and work with the latest changes, which helps avoid integration hell.

For the last while I have been involved with software deployment and I have written a couple of articles on the topic: Architecting for Deployability and Designing for Deployability.

I have encountered teams using ad-hoc build and deploy processes. I have observed such teams often experiencing production failures due to the incorrect implementation of changes into the production environment.

Communication

I see the other practices frequently mentioned by software consultants and software development books, but the practice of communication seems to receive less attention. However, I consider it equally essential. Frequent and open communication between team members, across teams, and across layers of the organization are critical to producing good software. Successful large open source software projects are excellent examples of communication in action: they have multiple mailing lists, standards for discussing changes and informing others of patches, and defect or task tracking tools for communicating issues and task assignments.

When communication is lacking or ineffective within a team, then I have observed work being done by one team member conflicting with or disrupting the work of other team members. Poor communication between teams can be more serious: teams working on interdependent applications or components need to keep the other teams informed and involved when the interfaces between them change. I have seen failure to do so lead to schedule slippages due to the additional work being identified too late, and even production failures.

You may feel that some of these practices are so obvious that they are not worth discussing. For every one of these practices, however, I have encountered individuals who did not see the value of the practice or who only adopted it grudgingly. If you or your team does not follow one of these practices, I strongly recommend giving it a try.

• A reporting tool that stores report definitions as binary files.
• A data modeling tool that persists models as binary files.
• An extract-transform-load (ETL) tool that writes the data to an object-oriented database.

I have witnessed the problems and struggles that occur when trying to use these tools as a result.

The first problem area is version control. Version control software such as Subversion supports concurrent modification of files without locking by handling concurrent changes to the same files through merging and diffing. This is handled automatically by the version control software for text files, but requires a custom diff and merge tool for each particular binary format you want to store. Development tools with binary file formats usually do not provide any diff or merge capability, and when they do it is built into the tool with no easy way to plug it into the version control software. For such tools, a frequent workaround is to use locks within the version control system to ensure that only one person is modifying each file at a time. Locks are generally painful to use: they are an extra step to perform for each edit, and they do not scale as the team grows in size or spreads across multiple locations.

The ETL tool I have encountered that writes to a database is even more difficult to use with version control. While you can take a backup of the entire database, it is not an effective way to do versioning as it is too course-grained. Fortunately, the tool provides the capability to export the transformations to individual text files and re-import them. So proper version control is possible, though awkward because of the extra steps necessary.

Another problem with tools using proprietary binary formats to store code is that sometimes you need to do something which the tool does not support. A common scenario is searching for uses of a particular artifact in order to perform impact analysis. Consider a real-life example: a database table is having a column removed. You need to verify first that this column is not being used in reports, extracts, transformations or loads. If the reporting tool or ETL tool does not provide an appropriate search function, then how can this impact analysis be done? If the tools write to text files, then you can do a text search on the file system, outside the tool, in order to find uses of the table column to be deleted. If the tools write to binary files, then this is not an option. If you instead need to rename the column and the tool does not provide a convenient way to do this across all reports or transformations, then it will likely be much easier to do a global search and replace across the text files. With binary files, you are stuck making the modifications within the tool.

A third problem with binary formats is that they restrict interoperability between tools and increase the risk of vendor lock-in. If you need or want to change tools, and neither tool has the appropriate functionality to convert to the other's format, then you are stuck if either tool restricts itself to a binary format.

Tools that manipulate graphics or diagrams often write to binary formats. In the case of bitmap images, a binary format makes a lot of sense, and I am doubtful you would ever want to merge changes to a single image. Diagrams or models that contain words as well as image information are candidates for doing merges or impact analysis, and therefore are better stored as text. This is possible with certain vector graphic formats such as SVG.

One general characteristic of the tools I have seen that do not write to text files is that they are marketed to non-developers. In some cases, the vendors of these tools claim that developer expertise is not necessary because you can create artifacts through graphical manipulation or wizards rather than traditional entry of code into an editor. It seems like this marketing mindset negatively impacts how well the tool accommodates typical development practices such as version control, automated builds, automated testing, and refactoring. Vendors seem to think that since developers are not necessary to use the tool, development practices are not necessary for the artifacts produced by the tool. I strongly disagree on both points.

Vendors may succeed in building tools that do not require the textual entry of code, but no matter what the user interface is, fundamentally the tool provides a means for a person to create a computer-readable specification for how to perform some operation. That is coding. Development practices such as version control, builds, deployments, and testing are applicable to all types of coding. The mechanism or user interface by which the code was created is irrelevant.

I am quite pleased that I was able to maintain my goal of publishing one article per week for the entire year. It has sometimes been challenging to meet this schedule since I have a full-time job and a family. Working four days a week instead of five has definitely helped. However, this fixed posting frequency has had some downsides. While I have no shortage of ideas for articles, I choose a topic based on what I am motivated to discuss, or based on issues I am currently dealing with. Some weeks I do not have a particularly strong motivation to write about a topic. I therefore have to push myself to produce an article which often I am not as happy with as my articles on topics I feel strongly about. Another downside is that writing an article a week does not leave enough free time for me to work on other projects. For example, I have an idea for a software utility that I would like to implement and add to my Software page, but this requires significant blocks of free time. Due to these factors, I am changing my publishing goals. My new goal is to publish a quality article at least every two weeks. I will also publish the articles immediately once they are written rather than always post on a fixed day.

One change to my website in the last nine months is the addition of advertising - specifically Google's Adsense ads. Adding advertising to a site can be controversial, so why did I do it? I do work full-time, so I do not need an income from this website. I do spend a considerable amount of time writing articles, however, and feel it is appropriate to be compensated for doing so. The deciding factor for me, however, was that it represented a learning opportunity I did not want to pass up. As I found out, I did have a lot to learn. Choosing ad placements was difficult: more visible positions above the fold would likely earn more, but would also be more intrusive for viewers. I wanted to maintain a good viewing experience for the reader, and did not want to plaster ads all over my site. Initially, I erred on the side of caution and only placed a small ad below each article. This placement earned very little revenue. I then used A/B testing to test a larger ad format in the same position. It performed about twice as well in terms of ePCM (effective cost per 1000 impressions), so I switched to that format. Later, I tried a new ad placement above the fold, and it performed even better. Currently, my ad placements are earning roughly five times my initial setup in terms of ePCM, which shows the importance of ad optimization. I have recently made other changes that I am looking forward to evaluating.

I have also changed the search functionality on this site. When I actually went to use the default WordPress search functionality, I was not pleased with the results that were returned (only posts are searched, not pages), and how the results were presented. I decided to switch to use Google's search, which not only returns better results in a format everyone is familiar with, but also can earn income if ads on the search result pages are clicked. I am quite happy with this change, and highly recommend it for other blogs.

To implement Google ads and search functionality required changing my custom WordPress theme. During my initial development of this theme, I had not bothered to check the code into version control. This caused me a few problems once I made changes after my website was live. One change made the site render badly in Internet Explorer, which I did not discover until after promoting the change to my live site, and without having the changes recorded in version control, I could not easily reverse the changes or determine what caused the problem. I therefore put my entire WordPress installation - not just my custom theme - into version control. This was very helpful not just for making changes but also for upgrading to new versions of WordPress. I always knew that version control was important, so I am not sure now why I did not use it from the start. At least now I have a better appreciation for its importance. For version control software, I use Subversion, and have written about my experience with it.

I regularly analyze my web server statistics, tracking metrics such as page views and referrer sources. I am pleased with how I was able to grow my traffic: my page views in December were more than the first four months of the year combined. I have used a few methods to promote my site including social bookmarking sites such as reddit.com, blog carnivals, and ProBlogger's group writing projects, which have all helped. One interesting lesson I have learned from my server statistics is that technical how-to articles that address a particular issue for a particular technology generally do better for drawing search engine traffic. One example of such an article is Tips for Using Log4j. Based on the search terms used, I can tell that people encounter problems working with a technology, search online for an answer, and come to my site. Partially as a result of such articles, my search engine traffic has increased by a factor of seven over the last six months.

Overall, I am pleased with this first year for my website and look forward to this coming year. I would like to thank all my readers, especially those who contributed via leaving a comment, linking to my site, or shared my site on a social bookmarking site. I would especially like to thank those of you who left donations. Your generosity and support is greatly appreciated.

One reason I like Subversion for personal use is that you can use it without running the server. The repository can be stored on your local file system and accessed by a Subversion client directly. To effectively use version control, I believe a graphical user interface is a necessity. Subversion only comes with a command-line interface that is great for automating builds, but insufficient for daily use. Fortunately, there is a great Windows SVN client called TortoiseSVN which integrates beautifully with the Windows Explorer shell.

As a Java developer, I've been spoiled by the exceptional CVS support in the Eclipse IDE. As good as TortoiseSVN is, it doesn't compare to having version control capabilities integrated into the IDE. Eclipse CVS support especially excels at doing difference comparisons and merges. So I was quite happy when I came across Subclipse, an Eclipse plugin for Subversion. I've only been using it for a short period of time, but its been great so far - just like Eclipse's CVS support. I've been able to use both Subversion clients on the same local workspace which I like since I can fall back to TortoiseSVN if Subclipse is missing functionality I need.

Setting up a Subversion repository is easy, but has a hidden catch. As with all version control systems, the first step is to create your directory structure with files that you want to version control. The obvious next step is to use this structure to create your repository. This will work for a while until you try to create your first tag (or branch). Then you'll discover you have nowhere to store the tag. Subversion treats tags and branches as simple copies of an existing directory structure that are themselves stored in the repository. To accomplish this, the Subversion documentation recommends setting up three root directories: /head, /branches, /tags. Let's assume your files to store in version control are all contained in a directory contained /projects. This would be stored directly under /head as /head/projects. If you create a tag called release-1.0, the corresponding version of your files would be stored under /tags/release-1.0/projects. If you have already set up your repository without this structure, it is easy to fix. Using the repository browser in TortoiseSVN, you can create and move directories directly in the repository, and these changes are themselves versioned so you can easily undo the change if necessary.

One issue I have with Subversion is that it creates .svn directories in each versioned directory for tracking the state of your local workspace. Version control software must store this information somewhere: I'd prefer to see it stored separately. On Windows, a logical choice would be an user's Application Data directory. Storing the .svn directories within the local workspace directories causes a few problems.

The main issue is renaming directories: if you don't use the Subversion client to rename a directory from /foo to /bar, then the new /bar directory will have a .svn subdirectory that thinks it is still directory /foo, and the Subversion client gets confused. In Java development, this is a problem when you want to rename a package. Using Eclipse to rename the package changes both the directory name and all the java files within the directory at once, but you need to notify the subversion client as well. If you are using Subclipse, then this happens automatically when you do the rename in Eclipse so there is nothing to worry about. If you are not using Subclipse, then it becomes a multi step process. First use TortoiseSVN to rename the directory, which updates the .svn directory contents. Then manually rename the directory back to the original name and use Eclipse to rename the package back to the new name, which will update the java files. Now both the Subversion local workspace and the java files consistently refer to the new name.

Another issue with storing .svn directories within the local workspace is that it becomes difficult to do manual operations on the local workspace. For example, if I want to set up a new Java project, the easiest way is to copy an existing one and then delete what I don't need. But if I do this, I'll copy all the .svn directories and confuse the Subversion client. If you want to zip up your workspace, it will by default include all the .svn directories and be roughly double the size it would otherwise be. One workaround for this problem is to use TortoiseSVN's export command which checks out files from the repository without creating a local workspace, which means that the .svn directories are not created, and that you cannot check in changes from these files since they are no longer connected to the repository.

Overall I am quite happy with Subversion, especially when used with the Subclipse plugin in Eclipse. If you use Subversion, what have your experiences been like? Any tips, tricks, or pitfalls to pass along?

What are the specific challenges facing developers maintaining software?

1. They probably did not originally develop the software, and may not even have access to the people who did. This is due to several factors. In some organizations the software maintenance group is separate from the software development group. Even when the same organizational group develops and maintains the software, it is common for the development team to be much larger than the number of people allocated to maintain the software, so the majority of the development team moves on to other projects. Another factor is that the lifespan of the software (five or more years) is usually longer than the lifespan of employees with the organization (average of three years).
2. They may need to make emergency fixes or release defect fixes quickly, while concurrently working on new features.
3. They must ensure that the software operates properly in its production environment, while developing and testing defect fixes and new features.

So how to address these challenges?

1. Make code readability a primary goal when writing code (after correctness). Rapid Development by Steve McConnell, page 255-256, relates an interesting experiment in which five development teams were tasked with writing the same piece of software. Each team was given a different objective (memory use, output readability, program readability, minimum statements, minimum programming time). Four of the five teams came in first place with respect to their objective, but no team came in first place in more than two of the objectives. The lesson: clearly define your objective, and make sure it is the right objective.

   Why is readability so importance for maintenance? Code you write is likely obvious to yourself, but try waiting a few months without looking at it, and then see what you think. And when it is another developer who has to look at the code, without the ability to ask you what you did, then readability becomes even more important. Since developers spend more time reading code than writing it, it makes sense to optimize the most frequently used path.

2. Use automated testing. Automated suites of tests using tools such as JUnit have been widely popularized in the last seven years, yet I still encounter developers not using this practice. In a discussion last year on automated unit testing, a senior developer stated that he did not see the need for such testing - he had no problems getting his code to work. While I doubt the validity of this reason - I find automated unit testing to be quite beneficial in catching defects in my code - more important are the developers who inherit this individual's code and must make changes. Automated testing is one of the best practices for reducing the risk of introducing new defects when changing the code. Another benefit of automated unit testing is that the unit tests serve as a form of documentation to new developers that shows how the code being tested is expected to be used.
3. Use version control software to its full potential. Most projects I have been involved with have used version control software, yet few have used it to the fullest extent possible. One example of this is the use of branching. It is a common practice in open source software development teams to use branching to manage changes to the currently released version of the software while implementing new features in the new version - exactly the situation faced in maintenance. Yet in commercial development, I have encountered many developers who either are completely unfamiliar with the concept of branching, or who are uncomfortable with performing branching and related activities in the version control software they use. For such developers, I recommend reading the book Software Configuration Management Patterns by Stephen Berczuk - it does an excellent job covering the concepts and recommended practices concerning the use of version control software (which is a subset of configuration management). I also recommend experimenting with branching with your version control software to become comfortable with the process. Create a test project with some files in your version control repository. Make a branch of this project, then modify files in both the original version and the branch. Then try merging / copying changes between the branch and the original version.

   Another case where version control software is not used to its full potential concerns which files associated with the application are actually being stored using version control. All projects using version control store the code, but many projects do not store other files associated with the application, such as web pages, online help, build scripts, database scripts, and technical documentation. Instead I have seen such projects use what I call the Common Server Directory anti-pattern. A set of directories on a file server is made accessible to the team, and everyone adds and modifies files in this common area. Sometimes an effort is made to track versions, via directory or file names, and sometimes not. Since a file system is not designed with the features of version control software, the end result is not surprising. The list of problems I have seen or encountered personally spans the entire spectrum: one person's changes overwritten by another, file corruption (on Windows servers), accidental moving of a sub-directory, confusion over previous versions of a file - who made which changes at what time, among others. Unfortunately, the solution is simple - use version control software - but hard to implement. Many of the other files associated with an application are created and/or owned by non-developers: web designers, business analysts, database administrators, testers, or technical writers. It is quite uncommon for such individuals to be familiar with the concepts of version control, let alone be comfortable using it. And such individuals are often in different groups than the development team, making it difficult to mandate the use of version control. If you had success implementing version control in this kind of situation, I'd like to hear how you accomplished it.

4. Ensure that the software is well-designed. Some qualities of a good design in the context of software maintenance are: as simple as possible, easy to understand, easy to make changes, easy to test, and easy to operate (i.e. easy to deploy and monitor for problems). Unfortunately, if you inherit an existing piece of software to maintain, then you will have to deal with the existing design, which may not be all that you'd like it to be. Even with well-designed software, as you add new features, you may find that portions of the original design are no longer appropriate, or that additional capabilities have to be included in the design. This is when the technique of refactoring is useful. Refactoring is a disciplined approach to modifying software code without changing its external behavior. A great book on this topic is Refactoring: Improving the Design of Existing Code by Martin Fowler. The more new features you need to add to an existing application, the more important refactoring becomes.

These recommendations should ideally be carried out when the software is first being developed, but it is never too late to start implementing these ideas. For those who have some say in the deliverables produced by a development project, please make sure items such as the automated test suite are included and are verified as part of the delivery. In one architectural review I did of a development project, I found evidence that JUnit unit tests were used by the developers, but they were not supplied in the release. On another maintenance project, the JUnit tests were present, but the configuration information necessary to connect to the database when running the tests was not documented or preserved in version control. I never was able to determine the configuration used (the individual able to run the tests had moved on before I joined the project), and I had to redesign the database connection code as a result.

This article is long enough that I deliberately did not go into details on how to implement each of the recommendations. If you are interested in reading more about any of these topics, let me know.