Tooling

WebSphere provides the wsadmin command-line interface for interfacing with WebSphere servers. Two scripting languages can be used: JACL (the default) and Jython, a Java-based version of Python. Invoke wsadmin with the argument "-lang jython" to use Jython. I recommend the use of Jython for two reasons. First, it is a more mainstream language. Second, the WebSphere Admin Console shows scripting commands in Jython equivalent to the GUI operations you last performed. See the screenshot below for an example:

You may think that since the Admin Console provides equivalent scripting commands to GUI actions that you could just copy and paste these commands into a script and you would be set. Unfortunately if you want the script to be re-runnable (on a given server) or reusable (on other servers), you will almost always need to make modifications. Certain operations, like creating a data source, will fail if there is an existing data source with the same name. So to have a re-runnable script you will need to add logic to your script to first detect if the data source you want to create already exists. Many commands use a hard-coded configuration id to refer to a particular item to be configured. This id generally consists of the item's name, directory path to the configuration file, name of the XML configuration file, and XML ID of the item within the file. While the first three are constant, the XML ID appears to be a numerically generated ID that will vary between servers. So you need to change the script to look up the configuration id.

The changes required to adapt the script for broader use are not trivial and require use of WebSphere APIs - in particular the AdminConfig and AdminTask objects provided to Jython scripts that are executed within wsadmin. The best tooling I have found for writing these scripts is to use Rational Application Developer which includes a Jython editor that includes wsadmin objects as part of the content assist. See the screenshot below for an example:

Examples

With the tooling in place let us look at some examples. The following scripts all assume a simple WebSphere topology of a single node with a single server that we are setting up - no clusters or multiple servers to worry about.
A number of settings require the configuration id or name of the server or node, so here's how to look them up:

nodeConfigId = AdminConfig.getid('/Node:/')
nodeName = AdminConfig.showAttribute(nodeConfigId, 'name')
serverConfigId = AdminConfig.getid('/Server:/')
serverName = AdminConfig.showAttribute(serverConfigId, 'name')

AdminConfig.getid() is one of the primary methods to use to look up configuration ids. But what is that "/Server:/" syntax used as an argument? That is called a containment path, which is a XPath-like syntax for looking up configuration ids. For more important details on containment paths and configuration ids read this excellent article by Vincent Partington.
As mentioned above, this only works when there is a single server defined on the node - with multiple servers, the call AdminConfig.getid('/Server:/') would return multiple server config ids separated by newlines. In this case if you want to configure a specific server you can look up the server configuration id by name as follows:

serverName = "testserver1"
serverConfigId = AdminConfig.getid('/Server:' + serverName + '/')

Having a configuration id is a start, but we still have not changed any settings. So here's a basic example of setting the maximum heap size to 1 GB:

maxHeapMb = 1024
jvmConfigId=AdminConfig.getid('/JavaVirtualMachine:/')
AdminConfig.modify(jvmConfigId, '[ [maximumHeapSize "' + maxHeapMb + '"]  ]')

This follows the standard pattern of first obtaining the configuration id and then changing the setting. I obtained the code for the AdminConfig.modify call from the Admin Console. But how did I figure out the AdminConfig call, in particular which containment path to use? This was painful initially to figure out. The hard-coded configuration id provided within the AdminConfig.modify call was "(cells/Node1Cell/nodes/Node1/servers/Server1|server.xml#JavaVirtualMachine_1183121908656)". The prefix "JavaVirtualMachine" on the XML id is the key type to use in the containment path.

Not all configuration commands require a configuration id. The AdminTask object typically takes arguments specifying the server and node name. Here's an example that prevents applications from accessing internal WebSphere classes:

AdminTask.setJVMProperties('[-serverName ' + serverName +
' -nodeName ' + nodeName + ' -internalClassAccessMode RESTRICT]')

The last step in any configuration script is to save the configuration changes. This is straightforward:

AdminConfig.save()

Resources

Here are some useful resources for this topic:

• Excellent article on the differences between configuration ids, containment paths and object names for WebSphere scripting.
• IBM article introducing scripting WebSphere using Jython
• Documentation on the WebSphere 7 Express sample scripts library
• List of commands available on the AdminConfig object
• Free online Jython language book
• Book published by IBM called "WebSphere Application Server Administration Using Jython"

Happy scripting!

Ant ships with two relevant tasks: <scp> and <sshexec>. Both of these tasks require the third-party library Java Secure Channel (Jsch) to provide a Java implementation of the SSH protocol. The jsch.jar file should be placed in the /lib directory of your Ant installation. While these tasks support both the SSH 1 and SSH 2 protocols, SSH 2 is preferred because it is much more secure. Use of SSH 2, however, requires Ant version 1.7.0 or greater.

Using the <scp> task appears at first glance to be quite similar to the <ftp> task. Here is an example:

<scp remoteTodir="${user}@${server}:${target.dir}"
  password="${password}"
  trust="yes"
  sftp="true">
    <fileset dir="${source.dir}">
      <include name="**/*"/>
    </fileset>
</scp>

This transfers files from ${source.dir} on your machine running the Ant build to the directory ${target.dir} on the machine ${server}. Despite the similarity to the <ftp> task, there are several significant differences:

• SCP transfers files as binary copies only and has no support for the ASCII transfer mode of FTP. This means that transferring text files between operating systems (Windows, Unix, Mac) can cause problems as the line ending characters vary across the three operating systems. The solution is to use the <fixcrlf> task to convert the line endings of text files before or after they are transferred.
• SCP transfers all the files in the fileset every time, whereas the FTP task supports transferring only newer files.
• SCP provides no mechanism for setting the file access permissions of newly created or existing files. On UNIX operating systems, SCP grants read & write access to the file owner and no permissions for the group or the world (other). These restrictive permissions are seldom appropriate – executable files should have the execute flag set and often the group needs read & execute permission. The solution I use to fix the permissions is to run a script on the target server using the <sshexec> task after the <scp> task to fix the permissions on the files & directories. This script is maintained as part of the code base and transferred to the target server via SCP along with the rest of the code. Below is an example of using <sshexec> to do this.

<sshexec host="${server}"
  username="${user}"
  password="${password}"
  trust="yes"
  command="perl -W ${target.dir}/bin/post_deploy_configuration.pl"
/>

One capability that the <scp> and <sshexec> tasks have that the <ftp> task lacks is the ability to use public key authentication instead of passwords. One significant advantages of using keys instead of passwords for an automated build is that keys do not expire, whereas passwords are often required to expire regularly. Keys are also much more secure in situations when the server or the network between the build machine and the server is at risk of being compromised.

The downside to using public key authentication is that some up-front work is required to set up and install the keys. The necessary steps are:

• Create a new key pair, either using the SSH software on your server or using an utility such as PuTTYgen. You should end up with a file holding the public key, a file holding the private key, and a passphrase for accessing the private key.
• Ensure the private key is stored in the openssh format, which is what the Jsch library used by the <scp> and <sshexec> tasks requires. PuTTYgen, for example, by default stores private keys in PuTTY's format but can convert a key to openssh format.
• Install the public key on the target server under the user id being used by the build. This involves copying the public key file into a specially-named subdirectory of the user's home directory and adding an entry listing the public key file to a special file within this subdirectory. The name of the subdirectory and special file depends on the specific operating system and specific version of SSH being used.
• Make the private key available to the build machine.
• Change your build.xml to include the following attributes for the <scp> and <sshexec> tasks instead of the password attribute: keyfile="${key.file}" passphrase="${passphrase}". Just like with passwords, do not embed the passphrase directly within build.xml. I store it in a property file which along with the private key is stored in a secured non-public location available to the build machine.

The <sshexec> task is limited to executing a single command on the server. The SSHTools project provides a more powerful <ssh> Ant task that supports automating an interactive remote session. Below is an example:

<ssh host="${server}"
  username="${user}"
  password="${password}"
  verifyhost="false"
>
  <shell term="vt100">
    <write>whoami</write>
    <read timeout="50000">${user}</read>
    <write>chmod -R u+rwx,g+rx ${target.dir}</write>
    <write>echo Done</write>
    <read timeout="50000">Done</read>
  </shell term="vt100">
</ssh>

I believe that producing good software is hard, and that we software developers need all the help we can get in developing software. I have put together a list of the top five practices that I find to be the most helpful in making it easier to produce good software:

• Automated Tests
• Version Control
• Refactoring
• Repeatable Build and Deploy
• Communication

These practices are generally applicable across the board: for new application development and ongoing application maintenance, for small teams and large teams, for custom enterprise software for a single customer and commercial shrink-wrap software, and for any language or technology. The specifics of how the practice is implemented will of course vary depending on the nature of the project.

Automated Tests

Automated unit tests provide a convenient way to test the software's functionality both during initial development and later as the software is maintained. This is especially essential during maintenance when the original developers are no longer around. Another benefit of automated tests is that they impose a design constraint upon the software that it be testable. This frequently helps improve the overall design of the software by reducing the amount of coupling between different components.

In order to be effective at finding defects, automated tests need to be written effectively. I have come across many examples of poorly written unit tests that are ineffective at finding defects. My article How to write good unit tests discusses this in detail.

Version Control

The use of version control software to manage changes to the software is indispensable for teams larger than one person. Some of the more important benefits of version control include helping coordinate concurrent changes to the code base, providing a history of changes, tracking the versions making up a release, and supporting parallel development on different versions of the software. I even use version control for my personal projects at home, where I am the sole developer. (Read about my experience using Subversion.)

Some of the problems I have observed when version control is not used are: each team member developing with a slightly different version of the software, resulting in chaos when it is time to deploy the software, and variations in the software installed in different environments, with no one able to explain for sure what the correct version is.

Refactoring

The design of a piece of software is a critical attribute for determining its quality. While good design does not imply good software, bad design generally does imply bad software. Refactoring is the key practice by which the quality of the design can be maintained and even incrementally improved as the software changes. Refactoring is a potentially risky activity because of the possibility of introducing defects. This risk is mitigated when the software has a comprehensive suite of automated tests.

Repeatable Build and Deploy

Every software team has some kind of method for building and deploying their software. I wanted to emphasize repeatability because that is the essential aspect of this practice. A build and deploy process that you can repeat consistently across team members, across environments, and across different versions of the software is critical to successfully shipping your software or deploying it into production. This practice only works effectively when version control is used to define the official version of the software being built and deployed.

A related aspect of this practice is keeping your active development mainline in good shape, which I define as having the code compile and pass all unit tests. This allows developers to reliably check out, build, and work with the latest changes, which helps avoid integration hell.

For the last while I have been involved with software deployment and I have written a couple of articles on the topic: Architecting for Deployability and Designing for Deployability.

I have encountered teams using ad-hoc build and deploy processes. I have observed such teams often experiencing production failures due to the incorrect implementation of changes into the production environment.

Communication

I see the other practices frequently mentioned by software consultants and software development books, but the practice of communication seems to receive less attention. However, I consider it equally essential. Frequent and open communication between team members, across teams, and across layers of the organization are critical to producing good software. Successful large open source software projects are excellent examples of communication in action: they have multiple mailing lists, standards for discussing changes and informing others of patches, and defect or task tracking tools for communicating issues and task assignments.

When communication is lacking or ineffective within a team, then I have observed work being done by one team member conflicting with or disrupting the work of other team members. Poor communication between teams can be more serious: teams working on interdependent applications or components need to keep the other teams informed and involved when the interfaces between them change. I have seen failure to do so lead to schedule slippages due to the additional work being identified too late, and even production failures.

You may feel that some of these practices are so obvious that they are not worth discussing. For every one of these practices, however, I have encountered individuals who did not see the value of the practice or who only adopted it grudgingly. If you or your team does not follow one of these practices, I strongly recommend giving it a try.

In my previous article on Designing for Deployability, I wrote about generating files per environment as one approach to handling environmental differences. EnvGen is designed to allow you to easily use this approach.

You can read more about EnvGen in the EnvGen Release Documentation.

Differences between environments that you are likely to encounter are:

• Database connections
• Third-party service connections (i.e. for a web service, or naming service)
• Logging settings
• Performance tuning settings (i.e. database storage options)
• Security settings (i.e. user ids, passwords)
• Directory paths (i.e. for installed programs or libraries)

Based on my experience, there are three design approaches to dealing with these environmental differences. They are discussed below in the order in which I feel they should be used: i.e. only use the second approach if the first does not work out, and only use the third when the first two approaches are not appropriate.

Use support provided by the platform - but only when it makes sense

The platform you are basing your development on - whether an application server, operating system, or set of language libraries - may have built-in support for dealing with certain environmental differences. Rather than building your own solution (which the other two approaches cover), it is often easiest to use the provided functionality. I have a number of examples involving a variety of platforms, including an example that shows why you should not necessarily use the built-in support if it has been badly designed.

The Java Enterprise Edition (Java EE) platform provides several options for dealing with environmental differences. One of the essential pieces is JNDI - the Java Naming and Directory Interface, which provides an environmentally-neutral way to lookup basically anything, ranging from simple strings to fully configured services. JNDI works great for looking up a DataSource as per the following sample code:

Context rootContext = new InitialContext();

String jndiDataSourcePrefix = "java:/"; // Varies by application server
String dataSourceName = "myDataSource";
DataSource dataSource = (DataSource) rootContext.lookup(
  jndiDataSourcePrefix + dataSourceName);

This not only hides the environment-specific details concerning the actual database connection, but also hides details concerning connection pooling which often vary between environments as well. The specifics concerning the database connection and connection pooling are specified within the application server for each environment. The code can therefore be promoted between environments without change.

Java EE also allows for simple strings to be stored in JNDI essentially like environment variables. The retrieval of these values is much like the retrieval of a DataSource as the following code shows:

Context rootContext = new InitialContext();
Context envContext = (Context) rootContext.lookup("java:comp/env");
String supportEmail = (String) envContext.lookup("support.email");

Unfortunately, this approach is not well suited to handling environmental differences. The values for such variables are specified in the deployment descriptor - ejb-jar.xml for EJBs or web.xml for web applications - as shown below:

<env-entry>
  <description>Support Email Address</description>
  <env-entry-name>support.email</env-entry-name>
  <env-entry-type>java.lang.String</env-entry-type>
  <env-entry-value>support@company.com</env-entry-value>
</env-entry>

The problem is that the deployment descriptor also contains important configuration information that does not vary per environment. Worse, the deployment descriptor file is bundled into the EJB jar file or WAR file for deployment. So how exactly can you specify different values for environment variables, without complicating your deployment process? You cannot.

The deployment descriptor represents a good idea but a flawed design. The creators of the Java EE specification envisioned multiple roles, including not just a developer role, but also an application assembler role and a deployer role. The idea was that the developer could specify the default value for the environment variable, and it could be overridden by either the assembler or deployer. However, the specification does not specify an easy way to do this override, and implies that the assembler or deployer would have to directly modify the deployment descriptor or use the proprietary administration interface of the application server as is necessary for configuring datasources and connection pools. The other flaw with this design is the idea of these separate roles. In practice, the developers are also the assemblers and often the deployers as well. When there are separate individuals doing the deployment, they typically know nothing about the application and could only override settings based on instructions provided by the developers.

The Java EE platform provides good support for data sources, but other types of environmental differences are really not supported well. I recommend not using the environment variable mechanism provided in Java EE, as it does not address the requirement of easily deployable software.

My next example involves Ruby on Rails, a web application framework for the Ruby language that has been receiving a lot of attention and hype in the last few years. One of the attractions of Rails is that it provides built-in support for many of the common features of web applications, including handling of environmental differences. Rails explicitly defines the notion of an environment via the RAILS_ENV environment variable, and even comes pre-configured with three: development, test, and production. Specifying environment-specific settings is extremely simple. There is a common file shared between environments (config/environment.rb), and one configuration file per environment (config/environments/<env>.rb). Since the configuration files are Ruby scripts, any type of setup can be done - you are not limited as in the case of Java EE deployment descriptor XML files. Rails is a clear winner over Java EE when it comes to deployability for multiple environments.

Parameterize and lookup differences by environment

If the platform you are using does not provide the support you need for multiple environments, then the next design approach is to implement your own support for multiple environments within the platform. While the specific mechanisms can vary, conceptually such support requires a parameter representing the environment (like the RAILS_ENV environment variable), and a lookup mechanism to retrieve environment-specific values based on this parameter (like the config/environments/*.rb files in Rails). The lookup mechanism can be as simple as settings in a property file named after the environment, or can involve properties in a database table keyed by the environment. A more sophisticated mechanism is to use a configuration interface with a factory to create the appropriate environment-specific subclass based on the environment. This allows for any type of setup to be implemented, rather than being limited to strings or other primitive types.

The catch with using this approach is that the parameter representing the environment must be handled using whatever support the platform provides for dealing with environmental differences. For a Java EE application this is not a big deal - there are several different options I have used. One approach is to store the environment in a special database table. Since Java EE handles the data source fine, the code can retrieve the data source and then query the environment table to determine the environment. Another approach is to use a Java system property or even an environment variable like Rails does. A third approach I have used is to have an environment specific directory (i.e. config/prod/) holding one or more property files or other resources. The classpath is defined within each application server to include the appropriate environment-specific directory. The code simply loads the property files or other resources from the classpath, which resolves correctly to the desired version of the files. This works especially well for log4j configuration files.

For platforms that provide absolutely no support for handling environmental differences, this design approach will not work. That is when the third approach becomes useful.

Generate per environment

This design approach is most appropriate when the platform provides no support for handling environmental differences. The most common example of this I have encountered is database DDL statements, which can have environment-specific storage settings but do not support variables that would allow one to parameterize these settings. If you want to fully script database changes, then it is necessary to have these DDL scripts be environmentally neutral. The solution is to use file generation to produce a different version of the script for each environment from a template using the appropriate values to substitute into the template for each environment.

I provide a software utility called EnvGen on my Software page that performs environment-specific file generation. EnvGen is an Ant task for generating different versions of the same file parameterized for different environments (i.e. development, test, and production). File generation is done using FreeMarker, a template engine with a full-featured templating language. You specify environment-specific properties in a CSV file (comma separated value spreadsheet). You can read more about EnvGen in the EnvGen Release Documentation.

For all of these approaches, no matter what language or platform you are using, the underlying concept remains the same: separate the settings and code that changes across environments from that which remains the same to achieve the goal of reliably and easily promoting the code into the production environment.

Why is deployability important?

Maintaining the proper operation of the production systems is a fundamental I.T. goal. The end users do not care how well the software works in the development or test environments. All the testing and other quality assurance activities in the world will not help if the software is not promoted properly into production and fails as a result. Ease of deployment is also important. If promoting changes is a cumbersome and labor-intensive task, people will be tempted to take shortcuts (i.e. skipping the test environment), and it will take longer to get changes into production.

How can you architect for deployability?

My first guideline is to minimize differences between environments. The more similar the environments, the simpler and more reliable it is to deploy the software. Each difference between environments is something that may trip you up when deploying if you have failed to properly address it. Environmental differences can also affect the accuracy of testing: what worked in one environment may not work in a different one. Unfortunately, it is not possible to eliminate all differences without eliminating all environments but one. Common unavoidable differences between environments include different servers, different hardware configurations, different databases, different URLs, and different security settings. When encountering an environmental difference, I ask the question "Is there a good reason for this difference to exist?". If not, then it should be eliminated.

One example from my own experience is a system deployed to Unix servers which had a standard directory structure except for the root directory that differed across environments. In my investigation I discovered that the reason for the difference was that the test server contained two separate test environments, each of which therefore required a different root directory. I considered this a valid reason given the existing hardware at the time. When the opportunity arose, I suggested that an additional test server be added and the second test environment moved to it to allow the paths to be standardized.

To handle unavoidable environmental differences, I recommend encapsulating them within the software to isolate these differences from the rest of the system. The specific mechanism will vary depending on the technology being used and the particular difference being addressed. I discuss some of these mechanisms in my article Designing for Deployability. The goal you should aim for is to be able to deploy the software using a simple, automated process. For deploying code, this typically means an automated script that takes the output of the build process and copies it to the appropriate location (usually on another server) for the environment you are deploying to.

In an earlier article on Deploying Application Changes, I distinguished between full deployments where the entire application or component is deployed as a unit, and partial deployments in which only the delta - the pieces that have changed - is promoted. As I discuss in that article, there are advantages and disadvantages to either approach. Within a single application, I prefer the full deployment approach. When multiple applications are involved, especially across multiple support or business units, I prefer to have the option of partial deployments. If separate applications are too-tightly coupled, then changes in one will require changes in the other, which then requires that both applications be deployed together. So high coupling between applications reduces the ease of deployability of each individual application. Clearly defined interfaces help minimize coupling, and if the changing application maintains backwards-compatibility of its interface, the other application will then not need to change, or can choose to change on its own time line.

Like most non-functional requirements, deployability is often neglected or overlooked during application development. Inexperienced or rushed developers can create systems that work great in development, but have many problems in the test or production environments due to issues with deployment. The absence of a defined and preferably automated deploy process or problems in testing due to environment differences are warning signs that more attention to deployability is needed.

My most significant growth was in the area of personal productivity: I read and implemented the organizational system described in the book Getting Things Done : The Art Of Stress-Free Productivity by David Allen. I have found this system very useful both at work and at home, and wrote an article describing my experience implementing the system.

This last year I switched to a four day work week, and have found it to be a significant improvement over the normal five days. I even figured out how to do this without a drop in pay.

One area of software development I have explored a lot this year has been change and release management: specifically the process of promoting application or infrastructure changes into a production environment. Both my experiences at work and my experiences running this website have made me appreciate the necessity of having a defined process. The amount of process that is necessary depends on the types of changes being made and the complexity of the environment. As I wrote in my article on deploying application changes, it is easier when the application is packaged and deployed as a single unit. When this is not possible - like for database changes - then more process is necessary. Adding too much process, however, can be just as harmful as having too little - a proper balance must be maintained.

Over this last year I have spent a lot of time thinking about application reliability. Ensuring a system is highly reliable is a surprisingly difficult task, and one of the major culprits is complexity. I have found myself arguing more strongly for simpler, more reliable solutions as a result. My most recent focus has been on error handling. My article on the fail fast and degrade gracefully approaches to error handling contains some earlier thoughts on this subject, but I have not yet written up my latest ideas. I have come to believe that the degrade gracefully approach, while more difficult to implement, is the best for creating highly reliable software. I have unfortunately encountered a third approach to error handling: ignore errors. This leads to silent, undetected failures whose negative effects go unnoticed for an arbitrary length of time.

I have also learned a number of lessons from running this website, including improving my knowledge of web design, promotion, and on-line advertising. I have found the process of writing my weekly articles to be a great learning aid. Writing forces me to reflect on and clarify my thoughts and ideas on a particular subject. It is no coincidence that I have provided a number of links above to articles I have written this past year. I tend to write about topics that are currently in my focus of attention, and the act of writing about them helps clarify and solidify the learning I have done.

ANT provides a FTP task. It requires the installation of two external libraries into ANT's \lib directory: Jakarta ORO and Jakarta Commons Net. The FTP task connects to a remote server via the FTP protocol and provides all the basic FTP operations: send files, get files, delete files, create directories, delete directories, and change permissions. For the full details on how to use the FTP task, see the ANT manual's entry on the FTP task.

For my website, I use the FTP task to transfer my server logs to my local workstation for analysis. Here is an example (using ANT version 1.6.5):

<ftp action="recv"
  server="ftp.server"
  remotedir="/logs"
  userid="my-userid"
  password="my-password"
  verbose="yes"
>
  <fileset dir="${log.dir}">
      <include name="*.log"/>
  </fileset>
</ftp>

This task transfers the files named *.log from the directory /logs of the server ftp.server to the ${log.dir} directory on your local machine.

At work, I usually use the FTP task for deploying application changes - sending a set of files built on my local workstation or development server to the test or production server. Here is an example:

<ftp action="send"
  server="ftp.server"
  userid="my-userid"
  password="my-password"
  remotedir="/incoming"
  depends="yes"
  binary="no"
  chmod="755"
>
  <fileset dir="${ftp.source.dir}">
      <include name="**/*"/>
  </fileset>
</ftp>

This task transfers all the files within the local directory ${ftp.source.dir} to the directory /incoming of the server ftp.server. Files in sub-directories are also transferred into corresponding sub-directories on the remote server, and the sub-directories are created if they do not already exist. Only new or changed files are actually transferred (depends="yes"). The files are transferred in ASCII mode (binary="no"). Assuming the remote server runs UNIX, the permissions of the files are set to 755 (chmod="755").

Using the FTP task to deploy a set of files has a number of limitations, especially if these files are organized into a hierarchy of directories. While the FTP task will automatically create sub-directories as required, I have not found a way to ensure that the required directory permissions are assigned for new directories (this assumes the remote server runs UNIX). In particular, the chmod attribute is only assigned to new files, not new directories. The only workaround I have found is to execute a script on the server that assigns the necessary directory permissions. Another limitation is that the FTP task will not create empty directories. If you want to create a particular directory structure on the remote server, then each directory you want created must have at least a single file in it. The workaround is to create a dummy file to ensure the directory is created.

Deleting obsolete files and directories on the remote server is also difficult for a couple of reasons. The FTP task does provide all the basic FTP operations, including deleting files and directories. But each is handled as a separate action which requires a separate invocation of the FTP task, instead of performing all the operations within a single FTP session. This is a minor inconvenience. The more significant limitation is in determining how to delete obsolete files and directories. The development build usually assembles a set of files to transfer to the server without caring about the individual files or directories - it just assembles a directory structure. Therefore, if certain files or directories are deleted or renamed, this just results in a new set of files without the build process knowing which are to be deleted. Therefore, automating the deletion of these obsolete files by listing each one is quite difficult. An easier approach would be to simply delete all the files and directories deployed to the server and then re-deploy the new set. This solution also has its problems. First, this requires recreating all the sub-directories, which as I explained above leads to the problem of ensuring their permissions are properly configured. Second, the directory structure into which files are deployed may include files created by server processes which cannot or should not be deleted, such as server logs or application data files. The workaround I use to deal with obsolete files and directories is to simple leave them in place - they typically have no impact on the operation of the system. But I am on the lookout for a better solution to this problem - if you know of any, please let me know via a comment.

Despite the limitations, I have been pleased with my experiences using ANT to automate FTP tasks. I encourage you to give it a try.

Update June, 2009: I have had questions about what version of the ORO and Commons Net libraries to use, as not all versions are compatible. I have had success with Ant 1.7, Commons Net 1.4.0, and Jakarta ORO 2.0.8.

This application deployment process must specify the set of application artifacts (i.e. executable code, database objects, files) that is deployed for a particular release - what I call the unit of deployment. There are two basic approaches to defining a unit of deployment.

The first approach is a full deployment: packaging together the entire application as a single unit. As a Java developer, this seems like a natural approach: the compiled Java class files along with other application files can be bundled into a single JAR, WAR or EAR file. A typical enterprise business application, however, consists of more than just the code. For the database in particular, deploying all the tables and views for the entire application when doing a release is not practical. Therefore, the second approach to defining a unit of deployment is a partial deployment: only include those application artifacts that have changed. The remainder of the artifacts are left unmodified in the environment being deployed to. For the database, this means that only those tables or views that have been modified need to be included in the deployment.

These two approaches each have advantages and disadvantages. The following criteria are useful for determining which approach should be used in a particular situation:

• How confidently can you track the application changes? If it is hard or error-prone to track changes to artifacts, then a full deployment is safer. With a partial deployment, you need to know that all relevant changes made in the development environment are being promoted to the test environment, and then from the test to the production environment. Missing one or more artifacts in the deployment to production can cause defects or even a complete failure of the application.
• How much manual effort is required to do the deployment? Manual effort increases the likelihood of a failed deployment due to human error. A full deployment is easy to automate and should require little if any manual effort to maintain, since you always just grab the entire application. A partial deployment can be automated for a single change, but it is hard to have a reusable automatic deployment mechanism since the set of artifacts that change will vary from release to release. Such a mechanism needs to be able to determine automatically what has changed and deploy just those pieces. A good example of this is the automatic update mechanisms found in operating systems and applications - they only download a patch containing the changes rather than an entire copy of the updated application.
• How much time is required to do the deployment? The longer it takes to deploy a single artifact, the more important this criteria, and the more valuable the partial deployment approach becomes. Database changes are the best example: to drop and recreate every table in the database for a change would require a full backup and restore of the data, which will often take prohibitively longer than simply updating the few tables that actually changed. Online application updates are limited by network bandwidth, so they also benefit from a partial deployment.

My preference is for full deployments. Unless circumstances require a partial deployment, I feel that the default choice should be to deploy everything. Unfortunately, this is seldom possible in a complex enterprise environment. Many enterprise applications consist of a collection of components or services such as web application, database, batch processing, scheduling, reporting, and web services. Due to organizational or logistical reasons, it is seldom possible to do a full deployment of all of these components in a single release. At the application level, therefore, a partial deployment is often done. It is at the component level where it is easier to achieve full deployments of the individual components.