• Documentation is more effective than code at communicating high-level or cross-cutting design and operational concerns.
• Larger organizations or distributed organizations cannot rely on face-to-face communications or having everyone co-located in one room so documentation has a role to play in knowledge transfer and communication.
• Documentation can target a non-developer audience, such as business representatives.
• Documentation helps protect against team turnover, which while a bad practice is not uncommon when using a vendor for development or maintenance or using separate development and maintenance teams.
• The act of creating documentation helps clarify thinking and identify gaps, thus functioning as a form of quality control.

This last reason is actually quite significant as it is often under-appreciated. In my experience it has happened quite often when I am working on design documentation for a body of code that I identify things that are sub-optimal, such as a badly named class or an unwanted dependency between components. This form of quality control is most valuable, however, when it uncovers gaps such as missing functionality or flawed design. Such gaps are very difficult for most other forms of quality control like testing or reviews to find.

The use of documentation templates makes it much easier to find these gaps by acting essentially as checklists of items to consider. I recently came across a great set of comprehensive templates covering all aspects of software development by Klariti . For a team or organization the price of these templates is ridiculously low - the full set of software development templates costs far less than a day's salary - and there are an assortment of free templates as well.

There are some potential drawbacks to avoid when using documentation templates. Some people have a tendency to want to fill in every section of a template or to use all available templates. This can waste a lot of time and effort. Focusing on identifying what is relevant and useful to document and doing only that is much more effective. Another drawback is that while the templates are in Office format (Word or Excel), you might be better served using a different medium such as a Wiki or a text-based format that is more friendly to version control. In these cases I would convert the templates to the desired medium.

Even if your team or organization has some templates, I think it would be beneficial to check out Klariti's templates and use them as a checklist to see if there is anything missing or needing revision within your own. That link again is: Klariti's software development templates.

I was tuning a batch processing application that received XML input data sets, each consisting of thousands of separate input records. The processing logic converted each input record into multiple Hibernate entities – as many as several hundred. This logic required a number of queries to implement - some to load related, preexisting entities, and others to verify consistency with existing data. This queried data would often be needed for multiple input records in the same data set. Based on this, I decided to use a single Hibernate session to process the entire data set, committing after each input record but keeping the session open to be able to make use of cached entities for subsequent processing.

When initial performance tests were carried out, they showed a disturbing trend: the processing time required per input record in the data set increased linearly. This meant that the total time required to process a data set increased exponentially with the size of the set! This is illustrated by the diagrams below.

An analysis of where the time was being spent showed that the majority of the processing logic required only constant time per record. Where was the extra time going? The culprit seemed to be the call to commit the transaction to the database. I knew that even a few hundred database insert/update statements would execute quickly in nearly constant time (databases are built to scale, after all). The actual database commit was equally speedy. Normally by default I assume that network calls will be the source of performance delays. But in this case this assumption appeared to be incorrect.

So what exactly was happening when I committed the Hibernate transaction, before the calls to the database? Hibernate's first step is to perform a flush to write all entities with changes (called dirty entities) to the database via insert/update/delete calls. How exactly does Hibernate determine which entities are dirty? For loaded entities Hibernate uses byte-code instrumentation to add logic to track when entities become dirty. But my scenario involved new entities, for which Hibernate could not work its magic. So on each flush Hibernate scanned the fields of each entity to see if there were changes. A linearly-increasing number of entities naturally led to a linearly-increasing time per flush. To make matters worse, Hibernate's flush algorithm apparently has a performance problem when dealing with cascaded collections, which I was using in my scenario.

The solution to my performance problem was to evict all the entities from the session after committing, thus making all entities detached, and then reattaching to the session the few entities I reused in subsequent processing.

This article is one of a series on Hibernate Tips & Tricks.

1. Thou shalt not break the build

This is the first commandment because breaking the build causes immediate problems for the rest of the team. Anyone updating from version control repository will have a non-functional local copy of the code and will be unable to continue with their work until the problem is fixed. A broken build also causes issues for others wanting to commit changes: they cannot verify that their changes do not break the code base. Doing an unrelated commit on top of a broken build risks introducing new breaks that compound the difficulty in getting the code base back to working order. (This exact problem happened quite recently on my project.)

The general principle is to not introduce problems in the code base that interfere with other development activities. Each team needs to have a specific, unambiguous definition. As a minimum standard I require code to always compile and pass 100% of the unit tests. While I prefer that automated integration and functional tests always pass, they are slower to run and depend on external systems, so pragmatically the occasional failure can happen.

An easy way to make your definition of a broken build explicit is to define an automated build process that performs this check. This build process can be executed by the developer prior to committing a change, and can also be configured to execute on a continuous integration server immediately after a change is committed to ensure the code base is not broken.

I like having a penalty in place for those that break the build. It can be a small monetary fine (that goes into the team pot for the next social event), receiving a mascot of shame, or being assigned an unpleasant task.

2. Thou shalt put everything in version control

All code and all related files must be in the version control repository. A new developer or a new instance of a continuous integration server should be able to check out a copy of the software and build a complete release from it.

Too often I have encountered code bases where some key files are not included. I have seen a number of projects not include third-party libraries in version control. This might be deemed acceptable if an enterprise library repository is used (using a tool such as Ivy), but in the cases I have seen the code base required a directory of library files to exist on each developer's local workstation without even a definition of the required versions of the libraries being used. Other common omissions include build scripts, configuration files, and documentation.

3. Thou shalt use the version control repository as the source of truth

The version control repository is the official source of truth for all versions of the software. Source code distributions may be created and distributed via shared network drives or via the web, but the source of truth remains the repository.

If multiple branches are being used within the repository then the purpose of each branch must be clearly defined. The typical approach is to use the trunk (mainline) for ongoing development work, and create branches for releases. So the trunk is the source of truth for the software overall, while each release branch is the source of truth for a particular release, and in particular the fixes and/or patches made for a release. Distributed version control systems such as Git, which in essence treat each local workspace as a separate branch, still need to follow these same rules to avoid complete chaos. (For example, running a continuous integration build requires at least one official branch to build from.)

The corollary to this commandment is that changes to source code not in the repository do not officially exist. Committing changes into the repository is therefore a critical part of doing development that is reflected in the next commandment.

4. Thou shalt commit thy changes daily

The principle behind this commandment is that the effort and difficulty involved in integrating separate changes increases non-linearly with the size of the changes, so it is best to integrate changes as often as possible. (This is an instance of the lean principle of flow which dictates minimizing large batch sizes and work-in-progress.) Requiring daily commits ensures that conflicts or duplication of effort are found quickly.

There are other benefits of daily commits. It forces developers to decompose their work into smaller, incremental pieces of work and ensure the quality of that work (since it cannot break the build as per commandment number one). Developers are more likely to use step-by-step refactoring rather than big-bang changes that are much riskier (and more likely to negatively impact the rest of the team). Daily commits provide a visible sign of progress per developer that can be used to assess when a developer is stuck or sidetracked (the commits tend to stop).

5. Thou shalt treat thy version control system as mission critical software

I have seen more than one company which did not treat their corporate version control system as mission-critical software. In one case the version control server was classified as a under-powered development server that could be bounced at any time or upgraded without prior testing. In another case the source code repository did not have robust backup and restore mechanisms in place.

If the version control server goes down it is safe to assume that regular development activities will start to grind to a halt within the day and any emergency fixes or scheduled releases will likely need to be delayed. It is a good idea to ensure that a version control server is available 24x7 with the ability to restore service within a half-day maximum. Why 24x7? Regular hours of development likely do not correspond to regular business hours (especially for emergency fixes), and scheduled builds typically run at night or the weekend.

Before getting into the specifics of Ivy, the first question to answer is why use Ivy at all? Why is managing build dependencies important? All the significant Java software I have worked on makes use of third-party libraries such as Hibernate for persisting objects to a relational database or JUnit for unit testing. Both the IDE projects and automated builds for these pieces of software have the common need to deal with these dependencies, most frequently by including them on the classpath for either compilation or at runtime.

These dependencies can be handled manually in an ad-hoc manner. This is what I have done in the past and what I have frequently observed to be done by other teams. But there are significant advantages to using a tool such as Ivy to manage these dependencies:

• Explicit tracking of the versions of the third-party software being used. Too often I have come across projects with a library directory holding a collection of third-party jar files without any information as to the versions being used. Ivy supports defining dependencies explicitly by version number. If dependencies are defined more loosely, such as via a version range (e.g. version 1.0+), then Ivy can generate a report listing the specific versions used. Knowing the version of dependencies is important when trying to resolve issues or make use of the latest features.
• Management of transitive dependencies. If your software depends on third party software that itself has dependencies on other software, Ivy can automatically pull in these transitive dependencies. When multiple dependencies themselves depend on different versions of the same third-party software, Ivy can automatically resolve the conflict. An example will help illustrate this. Let us say that your software depends on two other libraries B and C. B in turn depends on D version 1.0 while C depends on D version 1.1. Ivy will pull in B, C, and D v1.1 and leave out D v1.0.
• Standardized approach to storing dependencies. Ivy uses one or more repositories of software as the source for obtaining dependencies. Each piece of software in these repositories is characterized by a standard set of metadata: the key attributes are organization, module, and revision. Using a public repository that the third-party software you require is updated within makes it trivial to upgrade to a new version: just change the version you depend on, and Ivy can then automatically download the new version. You can also use an enterprise repository to publish your own software into in order to facilitate reuse by others.

There are a number of decisions to be made when using Ivy. In the remainder of this article I provide my experience and recommendations regarding these decisions.

Building Classpaths

There are two main options for building classpaths within your Ant build. The first is to use the Ivy CachePath Ant task to build an Ant path for your project's dependencies that references your local Ivy cache of dependencies. This nicely avoids any duplication of dependency information within your Ant build file. Unfortunately, the same cannot be done for your IDE classpath. Maven's solution is to generate the necessary IDE project files (e.g. Eclipse's .classpath file). The tooling support for Ivy to generate IDE files appears weak (I did not try it myself). Another limitation of this approach is that you will not have the libraries required for your project checked into version control. (You could, however, check an organizational repository into version control separately.)

The second option is to use the Ivy Retrieve Ant task to copy dependencies into your project workspace. You can specify the directory structure and file naming convention to use. I recommend copying the jars into a single lib directory (or one directory per Ivy configuration) and do not include revision information in the file names. This allows the classpath to be easily built in Ant using a simple fileset – e.g. <fileset dir="lib" includes="*.jar"/>. The IDE classpath must be manually maintained but at least keeping files revision-agnostic avoids needing to update the IDE when a dependency is upgraded. Because this lib directory is located within the project you can manage it in version control along with the rest of your project. One drawback of doing so, however, is that this lib directory full of jars now resembles a manual solution to maintaining dependencies so there is a risk that developers unfamiliar with the build or Ivy will directly add libraries to this lib path. I recommend using the retrieve task's option sync="true" to remove anything not specified as a current dependency.

I prefer the second option, partly because I am a fan of putting all project-related artifacts including third-party jars into version control, and partly because it makes it easier to manually manage the classpath within the IDE.

Using Repositories

Ivy requires at least one repository to be defined to obtain dependencies from and provides a number of options for doing so. One option is to use one or more public repositories: third-party repositories available over the Internet. Ivy provides built-in support for two: the public Maven repository, which is probably the largest and best known repository, and IvyRep, which only provides Ivy metadata for dependencies defined elsewhere (typically in the Maven repository). I recommend against using IvyRep: it appears it is no longer updated and I encountered errors resolving dependencies from it. The Maven repository seems like the best place to obtain third-party open source dependencies. You can use the site mvnrepository.com to search for the metadata for the dependencies you are looking for.

The Maven repository does have its share of problems. Dependency metadata does not always match the directory structure or is plain wrong. For large projects like Hibernate or Spring with many dependencies, I was unable to resolve the complete set of their dependencies out of the Maven repository. I have also heard about and observed stability issues: sometimes downloads of dependencies fail.

Since I believe that builds must be reliable and repeatable I feel that relying solely on a public repository is a bad idea. The alternative is an organizational repository, which could be defined for the team or project or spanning the entire enterprise. In either case, Ivy supports a number of options for defining home-built repositories with my favorites being the local file system (typically on a shared network drive) or SFTP/SSH to access a remote file system. Ivy also supports defining a chain of repositories, so you could define an organizational repository first and then a public repository second.

The approach I prefer, however, is to only use an organizational repository for builds and use public repositories solely to install dependencies into the organizational repository. The Ivy Install Ant task can be used to do this. It supports automatically installing transitive dependencies, which is very convenient but comes at a price. By default the install task fails if the dependency you are trying to install already exists in the target repository. This seems like reasonable behavior: you do not want to modify dependencies once they are installed. When combined with transitive dependencies, however, this is no longer useful: it is very likely that modules will have one or more common transitive dependencies, so when trying to install the second module, it will fail because it tries to transitively install a dependency that has already been installed. The workaround is to use the overwrite="true" option. I would prefer instead to have an option on the install task to ignore dependencies that are already installed.

When specifying a repository you must define the pattern used for the directory structure and file naming convention. My preferred pattern is: ${ivy-repository.dir}/[organisation]/[module]/[revision]/[artifact](-[classifier])-[revision].[ext]. Making the revision into a sub-directory allows Ivy to use atomic publishing. Specifying the optional classifier was required for certain dependencies that define, for example, source code and javadoc artifacts as well as the compiled code – without the classifier specified, all three artifacts resolve incorrectly to the same file (which Ivy fortunately identifies as an error).

Defining Dependencies

When defining dependencies for a module I believe that the general goal should be to use the minimum set of dependencies that are required. To do this effectively requires that you use configurations, which are basically defined sets of dependencies for a given module. Typical configurations include compile, runtime, and test. If you do not define the configuration(s) you want from a third-party module then Ivy defaults to pulling in all dependencies. For a third-party library like Spring which defines a large number of optional dependencies (in a configuration appropriately called "optional") this will result in pulling in a large number of jars that you simply do not require to compile or even run your code.

Should you explicitly define all your dependencies or automatically pull in transitive dependencies? One of the whole points of using a dependency management system like Ivy is to handle transitive dependencies, so I am a fan of using them. If you want to be aware of all the dependencies that are pulled in, you can produce a report using the Ivy Report Ant task that lists all the resolved dependencies for a module.

Conclusion

If you use Ant for your builds then I highly recommend using Ivy to manage dependencies. For an example Ant+Ivy build check out the Java Examples project available for download from the Software page.

FindBugs is a Java static analysis tool that scans compiled java code for potential defects and bad programming practices. Think of it as the Java compiler on steroids: it operates in roughly the same fashion but reports on a much larger set of errors and warnings. Static analysis makes a great complement to automated unit tests. Unit tests require effort to write, targets a specific piece of code, but can verify application-specific functionality. Static analysis requires no effort to write (beyond initial setup), targets the entire code base, but can only verify general code constructs. At least, that's the theory. How does it work in practice?

FindBugs supports a number of different ways of being used: command line, Swing GUI, integration into automated builds (i.e. via an Ant task and Hudson plugin), and Eclipse plugin. I decided to go with the plugin, and installation was as easy as adding the update site http://findbugs.cs.umd.edu/eclipse and installing it. Well, actually there were two gotchas. First, you need to be running Eclipse version 3.3 or greater for the plugin to work – RAD version 7 will not work. Second, you need to fully restart Eclipse after installing the plugin. I made the mistake of choosing the option to activate the plugin without doing a restart, which left portions of the plugin not working. It also seemed like you must bring up the Bug Explorer and Bug Details views, then restart Eclipse, in order to get those views working properly.

I used the FindBugs manual to get started. I selected my Time Reporter project to be the first guinea pig and ran FindBugs on it. I keep my code well-polished and well-tested with most Eclipse warnings turned on so I was not expecting FindBugs to turn up anything major. As I scanned the relatively small list of issues (under 30), I was surprised to see an actual defect! It actually took me a few moments of staring at it to find the problem. See the screen shot below.

It turned out this same defect occurred elsewhere in the code. FindBugs also identified cases of bad error handling that I would classify as defects: I was improperly ignoring return values from method calls like File.delete() or File.mkdir(). Most of these serious issues were in test code rather than application code, which made me feel a little better (but not much).

I fixed all the issues reported by FindBugs that I agreed with, turned off one FindBugs warning I did not agree with at all, and was then left with a small number of false positives – incorrect warnings about code that was actually correct. My preferred approach to compiler warnings is to have none in the code base. This is based on the fact that if your code base has existing warnings that should be ignored then it becomes very difficult to tell when you write some code that produces a warning that should instead be fixed. People become blind to all warnings if there are always some present. Warnings should be produced as part of the developer's regular process (i.e. writing code) rather than requiring an extra step. It was trivial to configure FindBugs to run automatically, but how to get rid of the unwanted warnings?

Eclipse warnings can be eliminated by using the Java 5 annotation @SuppressWarning, but FindBugs does not appear to support this annotation (or if it does I could not successfully determine what text must be supplied to the annotation to ignore the warning). I found some hints on the web that FindBugs does have the ability to ignore specific warnings, but it appears this feature has only been implemented in the Swing GUI and not in the Eclipse plugin. After further investigation I found a solution. I exported the current set of project warnings to an XML file, and then configured FindBugs to use that XML file as a baseline of warnings to ignore. This causes FindBugs to only report issues not in the baseline, leaving me with an empty list of FindBugs warnings – for now at least.

One annoying limitation of the FindBugs Eclipse plugin is that all configuration can only be done on a per-project basis. Unlike most other Eclipse functionality with global configuration that can be overridden on a per-project basis, FindBugs must be enabled and configured individually for each project. There were a few other wrinkles with the FindBugs Eclipse plugin. My general impression is that the plugin is still in a beta state, lagging behind the functionality offered by the FindBug Swing GUI.

Next I tried including FindBug as part of a continuous integration build running in Hudson. It was fairly easy to configure the Ant build to execute FindBugs on the project: the only snag was needing to allocate more memory for the JVM. Installing and configuring the FindBugs plugin for Hudson was likewise straight-forward, and resulted in a nice set of pages for viewing the trends and details regarding the FindBugs warnings. The big issue came when I wanted to filter out (exclude) all the warnings I did not want to fix. After much investigation and trial and error, I discovered that if I used the FindBug Swing GUI to create an exclude list, I could then configure the ant build to use that list as an exclude filter. Using the filter created by the Eclipse plugin did not seem to work. The FindBugs documentation concerning this was rather poor, but I did see a statement admitting that filter support was a bit messed up across the various FindBugs tools, and it sounds like this area will be targeted for improvement in the next year.

Despite these tooling problems, the ease with which FindBugs finds defects makes it definitely worthwhile to use on all Java projects. When I turned FindBugs loose against the code base of a reasonably-sized production application, it found 5 definite defects, 14 cases that were either potential defects or extremely bad coding style, and many other warnings (I didn't bother to count, but probably over 100) that were worth investigating and often worth fixing.

My conclusion from this is that using FindBugs is definitely worthwhile. I plan to roll it out to all my Java projects and integrate it into the automated builds so that the FindBugs results are also available from the continuous integration server. If you plan to adopt FindBugs then I recommend checking out some FindBugs tutorials. If you want to promote the use of FindBugs to your coworkers or management then I'll point out that FindBugs is a corporate standard at both Google and eBay and eBay reports that "using 2 developers to audit/review FindBugs warnings was 10 times more effective at finding P1 bugs than using two testers" (http://findbugs-tutorials.googlecode.com/files/UFIA-intro.pdf).

Using Resume Calculator

Resume Calculator is a Java program that is executed on the command-line. A sample resume file is listed below:

<?xml version="1.0" encoding="UTF-8"?>
<resume name="Foo Bar">
  <entries>
    <entry id="1" summary="Zoo Project" interval="2006-2007"/>
    <entry id="2" summary="Team Yak" interval="May, 2007 - Present"/>
    <entry id="Baz Project" summary="Team Lead for Baz" interval="Jan, 2008 - Mar, 2008"/>
  </entries>
</resume>

Executing Resume Calculator against this file to calculate the duration of entries "1" and "Baz Project" produces the following output:

Resume Calculator
Copyright 2008 by Basil Vandegriend.  All rights reserved.

Executed on: 8-Dec-2008 6:29:45 AM
Working directory: C:\Dev\Java\ResumeCalculator\dist\release\example

Using resume "C:\Dev\Java\ResumeCalculator\dist\release\example\sampleResume.xml".
Calculating for "Foo Bar" the duration of the resume entries:
1: Zoo Project
Baz Project: Team Lead for Baz
Duration is 14.0 months or 1.2 years.

Finished execution successfully.

More details on how to use Resume Calculator and the full format of the XML file are specified in the user manual included with the distribution.

Technical Details

Resume Calculator uses JAXB for parsing XML (which I just wrote about). I used the Joda Time library for manipulating dates – it is far superior to the standard Java classes. The Joda library, however, does not include logic for calculating the duration of multiple intervals of time so I had to code that functionality myself. This proved to be an interesting algorithm to write – thank goodness for unit tests. The source code for Resume Calculator is included in the distribution if you want to learn more.

By default Ant writes informational messages to the console. While this is usually sufficient for a successful build, more detail is often useful in the case of failed builds. Relying on the console output for information about the build has limitations – there might be too much output to fix in the console's buffer, or you might clear or close the console. The solution is to write Ant's output to a log file. This can be done using the <record> task, which allows you to specify a log level of "verbose" or "debug" to provide more details than the standard "info" log level. This task should be the first one performed by the build for any target to ensure that a log is always produced. To accomplish this, I put the <record> task at the start of a target named "init" which I ensure is the first dependency of every other target. An example is given below:

<target name="init">
  <record name="build.log" loglevel="verbose" append="false"/>
</target>

Normally I prefer to use the "verbose" level for the log file, as the "debug" level provides so much additional detail that it is only really useful for debugging bad behavior within Ant code. Notice that I specified the option append="false" which ensures that at the start of the build any existing log file is deleted and a new one created. This makes it easier to see what happens for an individual execution of the build and avoids having the log become too large. This can be a problem, however, if you need to refer back to the log from a prior build to resolve problems or for auditing purposes. I have had situations where after a build fails I look at the build log for the problem, run some other ant target (such as clean) before I have finished resolving the problem, then realize that I have wiped the record of the problem.

One solution to this issue is to create a separate log file for each execution of the build. This can be done by using a build timestamp as part of the log file name, as shown in the below example:

<target name="init">
  <tstamp>
    <format property="timestamp" pattern="yyyy-MM-dd_HH-mm-ss"/>
  </tstamp>
  <property name="build.log.dir" location="${basedir}/buildlogs"/>
  <mkdir dir="${build.log.dir}"/>
  <property name="build.log.filename" value="build_${timestamp}.log"/>
  <record name="${build.log.dir}/${build.log.filename}"
    loglevel="verbose" append="false"/>
  <echo message="Build logged to ${build.log.filename}"/>
</target>

The log files are written to a buildlogs subdirectory to avoid cluttering the main directory. For some reason the <record> task requires an absolute path when specifying a directory, so the build.log.dir property is prefixed with ${basedir}. After setting up the logging, a message is written to the console displaying the log filename used to make it easier to look up the log file if a problem occurs. This approach will cause log files to accumulate within the log directory. If this becomes a problem, you could add some Ant logic to delete log files older than a certain date. I will leave the implementation of this as an exercise to the reader. Hint: use a <delete> task with a <fileset> that contains a <date> selector.

I spend most of my development time working within the Eclipse integrated development environment and have come to rely on several of its search features. One is the Search | References | Workspace action available for Java code. It searches the entire code base for uses of the selected language element, which could be a field, method, class, or annotation. I like to call this the Ctrl+Shift+G operation after its keyboard shortcut. I always use the shortcut rather than navigate the menu, so much so that I have told coworkers to "use the Ctrl+Shift+G feature", and then flounder trying to come up with the actual name of the action. Performing this operation has become as natural to me as scrolling up and down or using the Open Declaration action. (This uses the F3 shortcut, and yes I had to look up this action's name as well.) I use the Search | References | Workspace action a lot when I am planning to make non-backward compatible changes to a block of code. I will look at all the users of the code and determine how they will be impacted and whether the planned change will work for them.

Two other Eclipse search features I rely on are Open Type (Ctrl+Shift+T) and Open Resource (Ctrl+R). Both are navigational aids allowing you to look up a Java type (i.e. a class) or a resource (i.e. a file on the file system) by typing in the name. I consider these search features because as you begin typing a name, Eclipse dynamically lists possible matches – see the image below. You can even specify a search pattern using basic wild-cards. I use these features almost exclusively to find Java classes. As a result I seldom have to remember what package a Java class is in, and do not have to spend time navigating through the package hierarchy to find it.

The final Eclipse search feature I use frequently is the Search | File action that allows you to find files in your workspace containing specified text or Perl-style regular expressions. I often refer to this as searching the entire code base. I find this quite useful when working with non-Java code such as JSPs, SQL, and Unix scripts. I typically use this feature when I need to determine where a particular construct is used – for example, a database table that is being altered. The Eclipse Search | File functionality brings to mind equivalent tools that I use on Unix systems, especially the find and grep commands. The grep command in particular I find useful for extracting important information from large log file. These tools are even available on Windows systems via Cygwin.

Probably the most obvious search tool for developers is Google. While it is quite useful for looking up reference information on languages, libraries, or tools, I find it especially valuable for finding solutions to perplexing errors. There have been a number of times that I have entered cryptic error messages from the database or application server directly into Google and found a solution or workaround within a short period of time. I find it interesting that often these solutions do not come from the official support site for the application but from a third party forum or blog. Google in essence allows me to tap into the world's collective knowledge and experience shared across the Internet and extract the information I need.

Other developers have blogged about their use of search and how it helps them. Jeff Atwood has written about how the Windows Vista start menu lets you type to find what you want to run. Cedric Otaku has mentioned that his use of browser bookmarks has dropped nearly to zero since Google came along. In both of these cases, search acts as an aid to navigation and memory. Rather than remember where a program or website bookmark is stored in a hierarchical menu and then navigate it, just type it in and scan the list of search results.

For me, the effective use of search feels like an upgrade to my mental capabilities – as if I am accessing a larger mind. A significant portion of human cognition deals with pattern recognition and memory retrieval via associations, and this what search tools do. One analogy is that my personal memory acts like a CPU-based cache – it is quick to access, but cannot store everything. If I get a cache miss – a question that I cannot personally answer from my memory – then I can turn to search. It is a slower operation to formulate & execute the search and then find an appropriate result, but I am able to access the collective mind via Google or the entire code base via Eclipse. When I find an answer, it is uploaded into my cache (my brain) for further processing. One limitation with search is that you do not always find what you are looking for. It depends on whether the information is available and on how effectively you formulated the search phrase or pattern. It is annoying to have a search fail, but it is really no different from our own fallible memory recall.

The similarities between electronic search and our own mental capabilities suggests that search will be a significant component of human-level artificial intelligences. Witness the victory over humans of checker- and chess-playing programs based on their ability to search potential outcomes of moves. There is no need, however, to welcome our electronically-searching overlords quite yet. The effective use of search requires formulating search terms and analyzing results, which in turn requires analytical and problem-solving abilities that current artificial intelligences lack across broad domains of knowledge. Until that is achieved, we can focus on using the power of search to boost our own capabilities.

Ant ships with two relevant tasks: <scp> and <sshexec>. Both of these tasks require the third-party library Java Secure Channel (Jsch) to provide a Java implementation of the SSH protocol. The jsch.jar file should be placed in the /lib directory of your Ant installation. While these tasks support both the SSH 1 and SSH 2 protocols, SSH 2 is preferred because it is much more secure. Use of SSH 2, however, requires Ant version 1.7.0 or greater.

Using the <scp> task appears at first glance to be quite similar to the <ftp> task. Here is an example:

<scp remoteTodir="${user}@${server}:${target.dir}"
  password="${password}"
  trust="yes"
  sftp="true">
    <fileset dir="${source.dir}">
      <include name="**/*"/>
    </fileset>
</scp>

This transfers files from ${source.dir} on your machine running the Ant build to the directory ${target.dir} on the machine ${server}. Despite the similarity to the <ftp> task, there are several significant differences:

• SCP transfers files as binary copies only and has no support for the ASCII transfer mode of FTP. This means that transferring text files between operating systems (Windows, Unix, Mac) can cause problems as the line ending characters vary across the three operating systems. The solution is to use the <fixcrlf> task to convert the line endings of text files before or after they are transferred.
• SCP transfers all the files in the fileset every time, whereas the FTP task supports transferring only newer files.
• SCP provides no mechanism for setting the file access permissions of newly created or existing files. On UNIX operating systems, SCP grants read & write access to the file owner and no permissions for the group or the world (other). These restrictive permissions are seldom appropriate – executable files should have the execute flag set and often the group needs read & execute permission. The solution I use to fix the permissions is to run a script on the target server using the <sshexec> task after the <scp> task to fix the permissions on the files & directories. This script is maintained as part of the code base and transferred to the target server via SCP along with the rest of the code. Below is an example of using <sshexec> to do this.

<sshexec host="${server}"
  username="${user}"
  password="${password}"
  trust="yes"
  command="perl -W ${target.dir}/bin/post_deploy_configuration.pl"
/>

One capability that the <scp> and <sshexec> tasks have that the <ftp> task lacks is the ability to use public key authentication instead of passwords. One significant advantages of using keys instead of passwords for an automated build is that keys do not expire, whereas passwords are often required to expire regularly. Keys are also much more secure in situations when the server or the network between the build machine and the server is at risk of being compromised.

The downside to using public key authentication is that some up-front work is required to set up and install the keys. The necessary steps are:

• Create a new key pair, either using the SSH software on your server or using an utility such as PuTTYgen. You should end up with a file holding the public key, a file holding the private key, and a passphrase for accessing the private key.
• Ensure the private key is stored in the openssh format, which is what the Jsch library used by the <scp> and <sshexec> tasks requires. PuTTYgen, for example, by default stores private keys in PuTTY's format but can convert a key to openssh format.
• Install the public key on the target server under the user id being used by the build. This involves copying the public key file into a specially-named subdirectory of the user's home directory and adding an entry listing the public key file to a special file within this subdirectory. The name of the subdirectory and special file depends on the specific operating system and specific version of SSH being used.
• Make the private key available to the build machine.
• Change your build.xml to include the following attributes for the <scp> and <sshexec> tasks instead of the password attribute: keyfile="${key.file}" passphrase="${passphrase}". Just like with passwords, do not embed the passphrase directly within build.xml. I store it in a property file which along with the private key is stored in a secured non-public location available to the build machine.

The <sshexec> task is limited to executing a single command on the server. The SSHTools project provides a more powerful <ssh> Ant task that supports automating an interactive remote session. Below is an example:

<ssh host="${server}"
  username="${user}"
  password="${password}"
  verifyhost="false"
>
  <shell term="vt100">
    <write>whoami</write>
    <read timeout="50000">${user}</read>
    <write>chmod -R u+rwx,g+rx ${target.dir}</write>
    <write>echo Done</write>
    <read timeout="50000">Done</read>
  </shell term="vt100">
</ssh>

This plugin is useful if you have your website hosted by an ISP and need to determine the version or configuration of your web infastructure stack to resolve problems or determine the compatibility of 3rd party software.