Visualizing package dependencies can be useful for a number of reasons:

• Reverse engineering the architecture of an application when documentation is limited, out-of-date, or non-existent.
• Automatically generating diagrams, saving the time required to draw them manually.
• To understand impacts when doing large-scale refactorings or changing implementations of components.
• To determine how to extract functionality for reuse.
• To support good architecture / design as espoused by Robert C. Martin in his book Agile Software Development, Principles, Patterns, and Practices, such as by eliminating cyclical dependencies or unwanted dependencies.

Here is an example package dependency diagram generated by Graphviz:

The source code provided below that was used to generate this example diagram is available from my Java Examples project on GitHub or as a packaged bundle on my Software page. The code can easily be reused: change the directory of compiled code scanned by JDepend ("dist/classes"), and change the root package used to limit the scope of the diagram ("com.basilv.examples.packagediagram").

package com.basilv.examples.packagediagram;
import java.io.*;
import java.util.*;
import jdepend.framework.*;

public class PackageDiagramCreatorApp {

  public static void main(String[] args) {
    createPackageDependencyDiagram();
    System.exit(0);
  }

  public static void createPackageDependencyDiagram() {
    Collection packages = analyzePackages();
    StringBuilder builder = generateGraph(packages);
    generateImage("packages", builder.toString());
  }

  @SuppressWarnings("unchecked")
  private static Collection analyzePackages() {
    JDepend jdepend = new JDepend();
    try {
      jdepend.addDirectory("dist/classes");
    } catch (IOException e) {
      throw new RuntimeException("Error adding directory for JDepend to analyze.", e);
    }
    Collection packages = jdepend.analyze();
    return packages;
  }

  private static StringBuilder generateGraph(
    Collection packages) {
    StringBuilder builder = new StringBuilder();
    builder.append("digraph packages {").append("\n");
    builder.append("node [shape=box];").append("\n");
    builder.append("rankdir=BT;").append("\n");
    Set drawnDependencies = new HashSet();
    for (JavaPackage javaPackage : packages) {
      String packageNodeName = getGraphVizNodeForPackage(javaPackage);
      if (packageNodeName == null) {
        continue;
      }
      builder.append(packageNodeName).append("\n");

      @SuppressWarnings("unchecked")
      Collection dependencies = javaPackage.getEfferents();

      for (JavaPackage dependency : dependencies) {
        String dependencyNodeName = getGraphVizNodeForPackage(dependency);
        if (dependencyNodeName == null
          || packageNodeName.equals(dependencyNodeName)) {
          continue;
        }
        String dependencyKey = packageNodeName + "->"
          + dependencyNodeName;
        if (drawnDependencies.contains(dependencyKey)) {
          continue;
        }
        builder.append(packageNodeName).append(" -> ").append(
          dependencyNodeName).append(" [weight=4]").append("\n");
        drawnDependencies.add(dependencyKey);
      }
    }
    builder.append("}\n");
    return builder;
  }

  private static String getGraphVizNodeForPackage(
    JavaPackage javaPackage) {

    String rootPackage = "com.basilv.examples.packagediagram";
    String packageName = javaPackage.getName();
    if (!packageName.startsWith(rootPackage)) {
      return null;
    }

    return packageName.replace(".", "_");
  }

  private static void generateImage(String fileName,
    String graphVizDotFormattedGraph) {
    try {
      File graphFile = createFileWithContents(fileName
        + ".txt", graphVizDotFormattedGraph);

      // This requires the Graphviz software to be installed -
      // see http://graphviz.org/
      String imageFileLocation = fileName + ".png";
      Runtime.getRuntime().exec(
        "dot -v -Tpng " + graphFile.getName() + " -o "
          + imageFileLocation);

      System.out.println("Image file available at "
        + new File(imageFileLocation).getAbsolutePath());
    } catch (IOException e) {
      throw new RuntimeException("Error generating image " + fileName, e);
    }
  }

  private static File createFileWithContents(
    String fileName, String graphVizDotFormattedGraph)
    throws IOException {
    File graphFile = new File(fileName);
    FileWriter writer = new FileWriter(graphFile, false);
    try {
      writer.append(graphVizDotFormattedGraph);
    } finally {
      writer.close();
    }
    return graphFile;
  }

}

Context

This pattern applies when you need to process a significant volume of data but the processing can be done incrementally on small subsets of the data. A typical example is loading a list of entities and then iterating through the list to process each one. While the results (output) of processing can be combined across all the entities, it is important that the input to the processing only requires a small subset of all the data, and not the entire list of entities. A code example illustrating this problem context is shown below:

List<Entity> entities = loadEntities();
List<ProcessingResult> results = new ArrayList<ProcessingResult>();
for (Entity entity : entities) {
  ProcessingResult result = processEntity(entity);
  results.add(entity);
}

Solution

Reducing the memory usage in the above example is based on the observation that loading the entire list of objects to process can consume a large amount of memory and is not necessary since we only use one object at a time. So the solution is to stream - incrementally retrieve - these objects instead of loading them all at once. For the consumer of this data the only change required is to first obtain a reference to the stream such as an Iterable that incrementally fetches data. Updating our prior code example results in the following (changed lines shown in green background):

Iterable<Entity> entities = streamEntities();
List<ProcessingResult> results = new ArrayList<ProcessingResult>();
for (Entity entity : entities) {
  ProcessingResult result = processEntity(entity);
  results.add(entity);
}

Examples

The mechanism to use for streaming objects will depend on the source of the data and may require significant changes compared to a bulk load. Here are some specific examples.

Parsing XML

Parsing XML files using JAXB is a convenient approach for converting the entire file into a tree of Java objects, but it populates the entire tree at once. To instead stream such data use the SAX parser provided as part of the JAXP API. The SAX parser is event-based, which means that it iterates over the entities (and attributes) of your XML and for each item invokes callbacks you define.

Querying Databases using Hibernate

When using Hibernate to query for a collection of entities it is convenient to simply ask Hibernate for the entire collection. A typical example of doing this using the query by criteria API within Hibernate is below:

public List<Entity> queryData() {
  Criteria criteria = session.createCriteria(Entity.class)
  // Add appropriate restrictions
  // ...
  List<Entity> result = criteria.list();
  return result;
}

When the criteria returns a large volume of data, however, this approach will consume a high volume of data. Instead use the scroll method on Criteria to return a ScrollableResults instance that can be used to iterate through the results. If you prefer to not expose the rest of the application to Hibernate classes, you can wrap the ScrollableResults in a special implementation of Iterator (which I leave as an exercise to the reader). The revision of the above example using streaming looks like the following (changed lines shown in green background):

public Iterator<Entity> queryData() {
  Criteria criteria = session.createCriteria(Entity.class)
  // Add appropriate restrictions
  // ...
  ScrollableResults scrollableResults = criteria.scroll();
  Iterator<Entity> result = new ScrollableResultsIterator(scrollableResults);
  return result;
}

This scroll approach only works when all the data can be processed within the same database transaction since the Hibernate session must remain open for the ScrollableResults to be able to continue fetching data. If this is not suitable then another option is to load the data using multiple queries that each return a subset of the data. One common example of this is when displaying search results to an user. Rather than showing all the results (which may number in the hundreds or thousands) show one page at a time and let the user step through the various pages of results. Due to the frequency with which this occurs I refer to this solution as paging. To implement this in Hibernate using the query by criteria API is fairly simple:

1. Start by creating your criteria object and defining its restrictions as you normally would.
2. Apply an ordering to the criteria. It is best if this ordering is consistent, by which I mean that database updates or inserts between queries will not result in invalid or unexpected results being returned. This assumes each query for a page executes in a separate database transaction which provides no guarantees of transactional isolation for the group of queries as a whole. In some contexts, consistency is not required. If it is then I prefer to use an auto-incrementing surogate primary key as the field to sort by in order to achieve the highest level of consistency.
3. Apply restrictions to retrieve only the specific page. This is done using the methods setFirstResult and setMaxResults on the Criteria object.

Consequences

One potential consequence of streaming data is a reduction in performance because data is loaded piece by piece rather than in bulk. To mitigate this, the solution is use what I call loading sets: define subsets of the total data volume that are small enough to not impact memory usage but large enough to minimize performance impacts. Then load the data one set at a time. The consuming API does not need to change: it can still iterate or stream over each loaded set, and then fetch the next set once the current one is exhausted.

You must be careful, however, that the cost of these defect prevention practices does not become excessive. That would introduce a different type of waste – non-value adding process. The waterfall method of software development is an example of this. One of the principles behind waterfall is that careful requirements analysis and design will minimize downstream defects during coding and testing. Put another way, it is a good idea to understand what you need to build before you start building it. The problems with waterfall arise from going to extremes in applying this principle. Requirements analysis is done up front for the entire project as a big batch based on the theory that it minimizes rework due to future change, but in reality the constant pace of requirement changes plus the learning that occurs throughout the project will result in increasing amounts of change the longer the time spent doing requirements. In contrast, Scrum and Kanban apply this principle using a balanced approach – project level requirements are done at a high level, and the more detailed analysis is done on individual user stories just prior to implementing them. (See for example the article Scrum and CMMI – Going from Good to Great: are you ready-ready to be done-done?.)

In order to effectively adopt a defect prevention practice two pieces of information are needed:

1. Specific, actionable steps to apply the practice.
2. The expected benefit. What categories of defects does the practice intend to prevent? This helps determine when to apply the practice and helps to evaluate it after adoption to assess its effectiveness.

If we consider the idea of careful requirements analysis and design mentioned above as a prevention practice, the benefits are fairly clear - prevent requirement and design based errors - but specific actionable steps are missing so it does not qualify. (In fact, this is one of the contributing factors why waterfall projects can end up in the analysis paralysis anti-pattern.)

Now that the groundwork has been laid I can present some specific defect prevention practices. This is not a comprehensive list – many other practices are possible. The practices I have chosen to discuss are ones that I have used and am confident that they work.

Use Understood Methods Rule

The basic formulation of the rule is quite simple: when coding a method only invoke other methods whose behavior you clearly understand and are confident will work the way you want. I have written a separate article providing specific guidance on how to apply this rule.

This practice generally aims to prevent interface errors - which I define generally as defects between two separate pieces of code. Research suggests that a significant proportion of defects are due to these kinds of errors (See for example the paper An Empirical Study of Software Interface Faults.)

I find this rule particularly valuable when applied to the invocation of methods between classes and especially between components. In this case it helps prevent integration errors which are usually not caught by unit testing.

Design by Contract

The idea of design by contract is to precisely specify the behavior of methods to help ensure that they are invoked correctly by callers and that the callers receive the results they are expecting. This is done by precisely specifying the preconditions and postconditions of methods. The chief proponent of design by contract is Bertrand Meyer, whose book Object-Oriented Software Construction is a classic on the topic.

Method preconditions are conditions that must be true in order for the method to successfully execute and fulfill its postconditions. Preconditions are most commonly applied to method arguments. For example, a method to convert a string to a date might have the precondition that the string argument must not be null. Preconditions can also be applied to the state within the class or even external state. For example, a method to delete a particular object from the database might have the precondition that the object exists within the database.

Method postconditions are conditions that the method guarantees to be true after execution, assuming the preconditions are met. Postconditions are most commonly applied to the return value of methods, but like preconditions can also be applied to the state within the class or external state. Returning to the example of a method that converts a string to a date, such a method could have two postconditions. First, that the method will return a corresponding date object that is not null if the input is a string in a valid date format, and second that the method will throw a specified exception if the input does not correspond to a valid date format.

The combination of preconditions and postconditions forms in essence a contract between the method and its caller. The caller promises to fulfill the preconditions in exchange for the method guaranteeing that the postconditions will be met.

Despite the fact that the name of this practice contains the word "design", this approach does not require a separate up-front design of each method. The goal is to have a clear specification of behavior once the method is finished – how you arrive at it is not important to this practice. I tend to start with an initial idea for a method’s contract that I evolve as I write tests and implement the method’s logic using test-driven development.

There are several options for specifying pre- and post- conditions. Some teams rely solely on their automated unit tests to serve as the specification, but I prefer a more concise specification provided as part of the method definition. In Java I typically use JavaDoc to document pre- and post- conditions and programmatically check argument preconditions at the start of the method. I typically formally specify pre- and post- conditions only on methods that are intended for use by other classes or components. In Java, this is typically public and protected methods of interfaces and classes.

This practice is very closely related to the Use Understood Methods Rule, and they go hand-in-hand. Knowing a method’s pre- and post- conditions is necessary to fully understanding it. As I stated above, I tend to only apply formal design by contract to methods intended for use outside the class in question, which means this practice is really aimed at preventing integration defects.

Defensive Coding

Defensive coding is named after the practice of defensive driving and is based on the same mindset of expecting problems to occur and actively taking precautions to avoid them. Defensive coding is applied by adopting a language-specific set of idioms that minimize or prevent common coding errors when using the language. These idioms are often reflected in coding standards.

Here are some examples of defensive coding idioms for Java:

• When comparing if a variable is equal to a constant, put the constant first. This avoids a potential null-pointer exception (if the variable is null) by invoking the equals() method on the constant, which is never null.

  public boolean isAdmin(String userId) {
    String constant = "admin";
    return constant.equals(userId); // Instead of userId.equals(constant)
  }
  

• Always use braces to define a block of code for an if, else, while, for, or do statement, even if the block contains only a single line of code. This avoids the problem of later adding a second line of code indented to the same level as the first and mistakenly thinking it will invoked as part of the block.

  public void addOptions(String userId) {
    if (isAdmin(userId)) {
      addAdminOptions();
    } else {
      addRegularUserOptions();
    }
  }
  

• Use the Java 5 for-each construct rather than using a loop index variable to manually iterate through a list. This avoids the problem of having an off-by-one error in constructing the loop.

  public void processOptions(List options) {
    for (Option option : options) {
      option.process();
    }
  }
  

Defensive coding aims to minimize coding errors, both at the time of coding and in the future when the code is being modified by others. While these types of errors are typically easily detected by unit testing, I find that using these idioms (after the initial adoption) takes virtually no effort or thought on my part, making them literally a no-brainer to use.

Example-Based Requirements

The idea behind this practice is to express requirements as much as possible in terms of concrete examples rather than the generalized wording typically used in use cases and lists of business rules which is almost always ambiguous. I have written a separate article providing further details on example-based requirements which includes a specific example. :)

The practice of example-based requirements aims at minimizing requirement errors, particularly errors due to misunderstanding or misinterpreting. The examples should also be used as acceptance test cases, in which case they help detect design or coding errors (although unit tests should identify most of these first).

Conclusion

I encourage you to choose one or more of these practices to adopt in your current development work. There will be extra effort initially to understand and become comfortable with a given practice, but this will decline over time as you achieve mastery of it.

So I decided it would be an interesting challenge to write about my mental processes during design and coding that I hope will prove beneficial for you. I use a recent development task I worked on as a case study for examining my thinking. The task involved designing and coding a framework for generating a set of reports for an application. In my analysis I identify a number of mental 'stages' that I used, which together loosely comprise my mental process map. I first present a narrative of my thinking during this case study with references to these stages identified in bold and then provide some concluding thoughts.

Case Study Narrative

I start by uploading the problem domain into my working memory – reviewing all relevant requirements and any upfront or preexisting design. I am not trying to gain comprehensive knowledge at this point – there are many specific, incidental details that are not relevant to the big picture that I can safely ignore. I instead focus on the significant elements that will feed into my initial design work:

• Domain Model: What concepts or data does the system need to manipulate or store?
• Process Model: What operations or events occur? How do they make use of the data?
• Constraints: What are the main constraints with respect to what I need to build? What do I need to consider or watch out for?

The initial upload raises a number of questions, points requiring clarification, and ideas for improvement (of the requirements and existing design). I consult with the business analyst and business team, using face-to-face conversation if possible, to gain clarity. Simultaneously I am thinking about the key concepts that I will use in the solution to address the required functionality. I use these concepts to assemble an initial domain model and process model – mostly in my head initially, perhaps supported by some sketches or doodles on a few pieces of paper.

As I iterate between understanding and clarifying the requirements and developing the concepts and models, I begin to balance competing requirements through a series of trade-offs. The most frequent trade-off is between minimizing design complexity and development cost versus fully providing the requested functionality. Having face-to-face meetings with the business helps me identify soft versus hard constraints and requirements. Soft ones can potentially be discarded through negotiation while hard ones are mandatory and must be addressed.

At this point I feel comfortable about certain parts of the design and feel fairly confident it will work, while for other parts I am still left with an uncomfortable feeling that further thinking does not help alleviate. This pushes me into an exploration mode in which I start writing code for the pieces I am uncertain about. I do not try to write complete, production-quality code. I instead do what I call 'design-level' coding where I define interfaces or classes with important method signatures, but with no real implementations for the methods – perhaps just some pseudo-code. At this point I find it hard to do TDD on non-trivial methods as method signatures and even the classes and interfaces can change dramatically. I leave lots of to-do comments in the code about specific questions or issues regarding specific functionality or design elements that are not relevant for the big picture I am working on. What I am looking for is significant gaps in my solution that may require additional clarification of requirements, or further refinement of the concepts and models I came up with earlier.

Throughout the entire process and especially during these initial stages there are times when I need a mental shift. This is usually when I am undecided how to resolve a particular design issue or when I feel mentally fatigued. I use a number of different strategies. One is to simply change location – get out of my cubicle and walk around. Another is to change activities to work on something unrelated and mentally less taxing in order to recharge. For thorny design issues I find that sleeping on them is a great way of letting the subconscious work on the problem and help arrive at a resolution.

At some point I feel that I have resolved all of the big uncertainties so I begin converting my separate chunks of design-level code into a unified set of working production-quality code. I call this consolidation. I usually begin by doing a sweep through the design-level code to resolve the outstanding to-dos. This helps identify any outstanding requirements clarifications that I need. I then switch to coding the functionality class by class and method by method using mostly strict TDD. Development feels slow at first because I need to write many utility methods or helper methods (for either the production code or for the test code), or refactor them into existence out of duplicate code that I introduce. But using TDD gives me that satisfying sense of progress as I slam out one fully-tested method after another.

Once the first draft of the code is written I polish it to ensure a high level of craftsmanship. This involves aspects such as renaming classes, methods and variables to ensure good readability, refactoring to eliminate duplication, and commenting when appropriate to ensure good maintainability. For more information on why and how to polish code see my article Why you should polish your code.

After reaching code-complete on the functionality I switch to feedback mode. I have two goals. The first goal is to pass the code through as many quality checks as I can to identify and eliminate defects. This includes asking for a peer code review, reviewing the results of static code analysis tools, verifying sufficient coverage by automated tests, adding automated integration tests, and performing manual functional testing. This list does not include automated unit testing because I have already done this concurrent with the coding. The second goal is to put the code to use to identify functional gaps, usability issues, and operational issues relating to non-functional attributes such as monitoring / logging, error handling, and performance. This second goal is especially relevant for infrastructure code, where putting the code to use generally means coding business functionality that exercises the infrastructure. Although I go into the feedback stage with usually ~95% code coverage from my automated unit tests, I do expect to discover and fix a few issues. As I progress through the stage, the code gradually stabilizes. At the end, it has reached feature-done status which means I consider it production-quality code ready for final testing.

Discussion

The process I have described may seem like it consists of discrete steps with a linear transition from start to finish, but it is anything but that. Each 'step' is fuzzy, blurring from one to the next. There are multiple transitions going back and forth between steps. Different portions of the functionality can simultaneously be in drastically different steps – I might be polishing one class while in the midst of consolidating a second and in exploration mode for a third. I like to characterize the actual process flow as chaordic - a blend of chaos and order based on balancing creativity with discipline.

The process may give an impression of a big-design-up-front approach, which is inaccurate for two reasons. First, I consider coding to be an act of design, so I am really designing throughout the entire process. Second, I do believe in doing an appropriate amount of thinking and analysis (what some call design) prior to starting coding. The amount needed depends on the size and complexity of the problem to be solved and my current understanding of it. For simple, straightforward problems I may only spend a few minutes doing this, but those few minutes will include upload, trade-off, and exploration activities prior to diving into the coding

In conversations with experienced developers I have noticed some correlations between how they describe the way they develop and my process, but there are also differences. So I am interested to hear what you think of this process and how it may match or differ from yours.

I was tuning a batch processing application that received XML input data sets, each consisting of thousands of separate input records. The processing logic converted each input record into multiple Hibernate entities – as many as several hundred. This logic required a number of queries to implement - some to load related, preexisting entities, and others to verify consistency with existing data. This queried data would often be needed for multiple input records in the same data set. Based on this, I decided to use a single Hibernate session to process the entire data set, committing after each input record but keeping the session open to be able to make use of cached entities for subsequent processing.

When initial performance tests were carried out, they showed a disturbing trend: the processing time required per input record in the data set increased linearly. This meant that the total time required to process a data set increased exponentially with the size of the set! This is illustrated by the diagrams below.

An analysis of where the time was being spent showed that the majority of the processing logic required only constant time per record. Where was the extra time going? The culprit seemed to be the call to commit the transaction to the database. I knew that even a few hundred database insert/update statements would execute quickly in nearly constant time (databases are built to scale, after all). The actual database commit was equally speedy. Normally by default I assume that network calls will be the source of performance delays. But in this case this assumption appeared to be incorrect.

So what exactly was happening when I committed the Hibernate transaction, before the calls to the database? Hibernate's first step is to perform a flush to write all entities with changes (called dirty entities) to the database via insert/update/delete calls. How exactly does Hibernate determine which entities are dirty? For loaded entities Hibernate uses byte-code instrumentation to add logic to track when entities become dirty. But my scenario involved new entities, for which Hibernate could not work its magic. So on each flush Hibernate scanned the fields of each entity to see if there were changes. A linearly-increasing number of entities naturally led to a linearly-increasing time per flush. To make matters worse, Hibernate's flush algorithm apparently has a performance problem when dealing with cascaded collections, which I was using in my scenario.

The solution to my performance problem was to evict all the entities from the session after committing, thus making all entities detached, and then reattaching to the session the few entities I reused in subsequent processing.

This article is one of a series on Hibernate Tips & Tricks.

An immutable class (or object) is one whose state cannot be changed once the instance is constructed. Mutable objects do allow state changes, typically via setter methods.

Below is an example of such a mutable class:

import java.util.Calendar;

public class Order
{
  private Calendar date = Calendar.getInstance();

  public Calendar getDate() {
    return date;
  }

  public void setDate(Calendar calendar) {
    this.date = calendar;
  }
}

The public property date uses Calendar as its type. Calendar is itself a mutable class – it has methods such as setters that modify the state of the class. So what is the issue? Consider the following code:

  public static void exampleOne(Order order) {
    Calendar cal = order.getDate();
    // Is order past due?
    cal.add(Calendar.DAY_OF_YEAR, -10);
    if (cal.before(Calendar.getInstance())) {
      // Order past due logic...
    }
  }

Can you spot the problem? I must admit that I failed to notice anything wrong at first glance. Here is the same example instrumented with print statements. Can you predict what will happen when you run it?

import java.text.DateFormat;
import java.util.*;

public class ExampleTwo
{
  public static void exampleTwo(Order order) {
    print("Original order date = "
      + convertToText(order.getDate()));
    Calendar cal = order.getDate();

    // Is order past due?
    cal.add(Calendar.DAY_OF_YEAR, -10);
    if (cal.before(Calendar.getInstance())) {
      // Order past due logic...
    }
    print("Ending order date = "
      + convertToText(order.getDate()));
  }

  private static void print(String message) {
    System.out.println(message);
  }

  private static String convertToText(Calendar calendar) {
    return DateFormat.getDateInstance().format(
      new Date(calendar.getTimeInMillis()));
  }

  public static void main(String[] args) {
    Order order = new Order();
    exampleTwo(order);
  }
}

Below is the console output from executing this code:

Original order date = 8-Aug-2008
Ending order date = 29-Jul-2008

This clearly shows the problem: the date contained in the Order instance is changed within the exampleTwo method, which is likely incorrect behavior. The trap for a developer using the Order class (i.e. to write exampleTwo()) is that they would not necessarily consider or realize that changing the Calendar object returned by order.getDate() would change the value within the instance. The Order class is correct (no defects) but the implementation is unsafe because a mutable object is returned from the getter. How can this be addressed?

One solution for this particular example is to take the order-past-due check and turn it into a method on the Order class that does not change the date. While I would likely employ this solution, it unfortunately does not address the underlying issue. Even if the getDate() method is removed, the underlying problem can occur with the setDate() method. The following example demonstrates this:

import java.text.DateFormat;
import java.util.*;

public class ExampleThree
{
  public static void exampleThree() {
    Calendar calendar = Calendar.getInstance();
    Order firstOrder = new Order();
    Order secondOrder = new Order();
    firstOrder.setDate(calendar);

    calendar.add(Calendar.DAY_OF_YEAR, 10);
    secondOrder.setDate(calendar);

    print("First date = "
      + convertToText(firstOrder.getDate()));
    print("Second date = "
      + convertToText(secondOrder.getDate()));
  }

  private static void print(String message) {
    System.out.println(message);
  }

  private static String convertToText(Calendar calendar) {
    return DateFormat.getDateInstance().format(
      new Date(calendar.getTimeInMillis()));
  }

  public static void main(String[] args) {
    exampleThree();
  }
}

The console output is:

First date = 19-Aug-2008
Second date = 19-Aug-2008

The same instance of Calendar is added to both Order instances, so when the calendar's date value is changed for assignment to the secondOrder instance, it also changes within the firstOrder instance. This is likely incorrect behavior which occurs because the setter directly stores the provided mutable object. So we end up with two references to Calendar in each of the Order instances that point to the same instance. This is called aliasing.

By now you are perhaps convinced that getters and setters should never deal with mutable objects. It is unfortunately not that simple. Consider the following example:

public class Order
{
  private Customer customer;

  public Customer getCustomer() {
    return customer;
  }

  public void setCustomer(Customer customer) {
    this.customer = customer;
  }
}

public class Customer
{
  private String name;

  public String getName() {
    return name;
  }

  public void setName(String name) {
    this.name = name;
  }
}

public class ExampleFour
{
  public static void exampleFour(Order order) {

    print("Original order customer name = "
      + order.getCustomer().getName());

    String newName = "New name";
    Customer customer = order.getCustomer();
    customer.setName(newName);

    print("Final order customer name = "
      + order.getCustomer().getName());
  }

  private static void print(String message) {
    System.out.println(message);
  }

  public static void main(String[] args) {
    Customer customer = new Customer();
    customer.setName("Starting name");
    Order order = new Order();
    order.setCustomer(customer);
    exampleFour(order);
  }
}

The console output is:

Original order customer name = Starting name
Final order customer name = New name

In this case it seems perfectly reasonable to update the customer's name and have this change remembered within the Order instance. Before I explain the difference between these scenarios, I will show one more example involving a collection property:

import java.util.*;

public class Order
{
  private Customer customer;

  public Customer getCustomer() {
    return customer;
  }

  public void setCustomer(Customer customer) {
    this.customer = customer;
  }
}

public class Customer
{
  private List orders = new ArrayList();

  public List getOrders() {
    return orders;
  }

  public void addOrder(Order order) {
    if (order == null) {
      return;
    }
    orders.add(order);
    order.setCustomer(this);
  }
}

In this case, calling customer.getOrders().add(order) might seem reasonable but is not correct as this will not invoke the extra logic in customer.addOrder(order) to call order.setCustomer().

The following table summarizes the above discussion.

Given that the code for the Calendar property compared to the Customer property is basically identical, why should we have different expectations of their behavior? The reason has to do with the nature of the two types. Calendar is a Value Object - a simple object whose equality is based on its value rather than its identity. Changing a calendar means ending up with a new date that is not equal to the original. Customer is a Reference Object – an object with business meaning whose identity is the basis for equality. Two customers can have the same values but be distinct. Changing a customer is just that – a change to that customer that should propagate throughout the system. For a fuller discussion of value objects see page 486 of the book Patterns of Enterprise Application Architecture by Martin Fowler. This discussion includes a short section on the risk of encountering aliasing defects if a value object is mutable and recommends that they be immutable.

I agree with this recommendation and I think it should be a design principle for classes: class properties that are value objects should be immutable. Many typical value object types in Java such as String and Long are already immutable, but we have already seen that others such as Calendar and List are not. What are the options in such circumstances? I will address this question in a future article.

The source code listed in this article is provided in the Java Examples project which can be downloaded from the Software page.

In traditional software development methodologies such as the waterfall methodology, software design is an explicit phase that produces a design document as an output of that activity. The design is usually completed before coding starts. Jack takes a contrary view: the main thesis of his article is that "final source code is the real software design", and most of the article is dedicated to exploring the ramifications of this thesis. The main implication is that "programming is a design activity—a good software design process recognizes this and does not hesitate to code when coding makes sense. " And it is not just coding that is a design activity. "Coding is design, testing and debugging are part of design, and what we typically call software design is still part of design. " One may get the impression that Jack is just a cowboy coder who disdains the traditional notion of design. The following quote, however, shows that Jack values all types of design. "In software engineering, we desperately need good design at all levels. In particular, we need good top level design. The better the early design, the easier detailed design will be."

As Jack expands on the implications of the source code being the design, he clearly demolishes the assumptions behind a traditional waterfall approach to development by emphasizing the need for the design to evolve and be refined. "Eventually, we have to create the real software design, and it will be in some programming language. Therefore, we should not be afraid to code our designs as we derive them. We simply must be willing to refine them as necessary. ... All design activities interact. A good software design process recognizes this and allows the design to change, sometimes radically, as various design steps reveal the need."

Jack's follow-up article to his original 13 years later clarified his stance on design documentation beyond that of the code. "The source code may be the master design document, but it is seldom the only one necessary." But he also pointed out the flaw in insisting that this separate design documentation be a formal deliverable: "What approach they choose doesn’t matter; until someone starts insisting that these intermediate designs should be products in their own right. It’s the code that matters. If you get good code, does it really matter how it came about? If you don’t get good code, does it really matter how much other garbage you made people do before they wrote the bad code?"

These points I've quoted and almost of the articles ring true for me. I especially love the last quote – what truly matters is the code. When it is deployed into production, it does not matter how polished the design document was. In some of the formal, document-heavy client projects I have been involved with, the focus placed upon the project documentation is extreme, with the client in many cases critiquing even the grammar or punctuation within these documents. Meanwhile the code is produced with limited if any code reviews and inadequate testing. Recently on a project, after all the effort I and others put into the design document to obtain the formal sign-off by the client, I had to change part of my design after only a few days of coding. (And that is after I 'cheated' and coded a prototype during the design phase.) I discovered that another part of the design I was not involved with changed at the end of the design phase and invalidated one rather key assumption I made within my own design.

I find myself agreeing with Jack not just on the basis of his arguments, but also on the basis of my own experience. So why then are significant portions of I.T. still focused on the traditional waterfall / document-heavy approaches to developing software? Are there counter arguments I am missing? Please let me know by posting a comment, especially if you disagree with Jack and me.

• Ask people to help.
• Look at what people do.
• Learn from the facts you gather.
• Try it yourself.

IDEO designs an incredibly wide variety of products – corporate websites, hand-held electronics, clothing, business services, furniture, and more. With such diversity, it is no surprise that not all of the cards appear applicable to software development. I found that going through the cards and thinking about how they relate to I.T. to be interesting. I ended up classifying the relevant cards into three groups:

• Requirements: These cards provide ideas on how to gather or analyze requirements. This was the largest group by far – over one third of the total number of cards, and they spanned the four categories above. If you work in I.T., it may seem that calling tips for requirements design approaches is inappropriate. In software development there is often a divide between requirements and design. You do need to understand the client's needs and determine a solution that meets those needs, but an explicit separation in role between these activities can often hurt the final product. IDEO takes a holistic approach: determining requirements and understand the user is part of the design process and not a precursor to it.
• User Interface Design: About one-sixth of the cards presented ideas related to user interface design. These mostly fell into the Try category, with a few in Learn and Ask. Prototyping and testing of various sorts were reoccurring themes in over half of these cards.
• Software Design: Only a few cards seemed relevant to application architecture and software design. I initially found this surprising since I was expecting more. After further reflection, I realized that the commonly held understanding of design in the context of software development is very technical and narrowly focused. This 'technical' design activity (for lack of a better term) is necessary but not sufficient for creating a great piece of software. Other activities within the course of a development project that we in I.T. do not call design – activities such as requirements gathering, analysis, and usability testing – are all part of IDEO's holistic view of design.

In order to provide a concrete example of what the cards are like, I list in the table below a few of the cards I found particularly interesting. Each card explains not only a design method (the how), but also the reason for using it (the why). (Each card also briefly describes an IDEO project that used this method, but I do not list that below.)

Looking at these design methods and the many others listed in the set of IDEO cards makes me appreciate all that goes into a well-designed product, and inspires me to think more carefully about how I think about and approach design.

WordPress previously only supported categories, which I used on this site. WordPress supports associating a post with any number of categories, just like tags. So how do they differ? The first major difference is that WordPress expects categories to be predefined, whereas it allows new tags to be added without restriction. The second major difference is that the navigation element for categories is most frequently a list. Since a category list takes up much more vertical space than a tag cloud for the same number of elements, there is the expectation that the total number of categories will remain small. See the image below.

Comparing the two images, we see that the tag cloud actually takes less vertical space despite showing four times as many items. The tag cloud does use more horizontal space, but this is actually an advantage: the format of the tag cloud allows it to use almost all of the available horizontal space, unlike the category list. The tag cloud, therefore, uses the screen real estate more effectively.

Which format is more usable? The main goals of both the category list and tag cloud are to (1) provide an overview of what the site is about, and (2) help users navigate to articles of interest. I believe that the tag cloud does a better job of achieving both goals. Displaying so many more tags than the category list gives a much more complete overview of the content available on the site. There is a risk that too many tags will cause the more important content to be lost in the crowd, but this is offset by the use of larger fonts for more frequently-used tags. I suspect that navigation is generally easier for users looking for a particular topic because it is much more likely there will be a tag that corresponds to their topic. If there isn't, they do have more tags to navigate through compared to a category list, which is a disadvantage. But on average there are less articles per tag then there are per category, so they can try several tags fairly quickly. For myself, I often look up old articles while writing new ones, and I find it much easier to navigate to a specific old article via a tag than via a category. Why? I often have difficulties remembering exact what 'vague' category I filed an article under, while there is usually at least one tag I know is linked to the article.

This brings me to what I believe is the most significant advantage of tags over categories: tags correspond more closer to how our minds store and retrieve information. Categories imply a hierarchical, structured way of organizing information, with each post usually being filed in a single category. Tags imply a more arbitrary network of relationships between articles which supports multiple ways of categorizing the content. I have often struggled in the past trying to assign a post to a single category, especially when part of the post is about another category. Assigning multiple tags is much more natural in this situation.

I haven't seen any consensus on the web regarding whether tags or categories are better. Among popular social bookmarking sites Digg uses categories and Technorati uses tags. An article on Problogger about tags and categories and the resulting discussion revealed a diverse set of opinions; the article itself stated that tags compliment categories and both can co-exist on a site.

I myself am leaning towards eliminating the use of categories on my site and just using tags with the tag cloud. I would be interested in hearing your opinions on the matter. Do you prefer the tag cloud or the category list? Should I keep both? Should the tag cloud display less tags, or have a smaller size for the largest font? Please leave a comment and let me know.

Differences between environments that you are likely to encounter are:

• Database connections
• Third-party service connections (i.e. for a web service, or naming service)
• Logging settings
• Performance tuning settings (i.e. database storage options)
• Security settings (i.e. user ids, passwords)
• Directory paths (i.e. for installed programs or libraries)

Based on my experience, there are three design approaches to dealing with these environmental differences. They are discussed below in the order in which I feel they should be used: i.e. only use the second approach if the first does not work out, and only use the third when the first two approaches are not appropriate.

Use support provided by the platform - but only when it makes sense

The platform you are basing your development on - whether an application server, operating system, or set of language libraries - may have built-in support for dealing with certain environmental differences. Rather than building your own solution (which the other two approaches cover), it is often easiest to use the provided functionality. I have a number of examples involving a variety of platforms, including an example that shows why you should not necessarily use the built-in support if it has been badly designed.

The Java Enterprise Edition (Java EE) platform provides several options for dealing with environmental differences. One of the essential pieces is JNDI - the Java Naming and Directory Interface, which provides an environmentally-neutral way to lookup basically anything, ranging from simple strings to fully configured services. JNDI works great for looking up a DataSource as per the following sample code:

Context rootContext = new InitialContext();

String jndiDataSourcePrefix = "java:/"; // Varies by application server
String dataSourceName = "myDataSource";
DataSource dataSource = (DataSource) rootContext.lookup(
  jndiDataSourcePrefix + dataSourceName);

This not only hides the environment-specific details concerning the actual database connection, but also hides details concerning connection pooling which often vary between environments as well. The specifics concerning the database connection and connection pooling are specified within the application server for each environment. The code can therefore be promoted between environments without change.

Java EE also allows for simple strings to be stored in JNDI essentially like environment variables. The retrieval of these values is much like the retrieval of a DataSource as the following code shows:

Context rootContext = new InitialContext();
Context envContext = (Context) rootContext.lookup("java:comp/env");
String supportEmail = (String) envContext.lookup("support.email");

Unfortunately, this approach is not well suited to handling environmental differences. The values for such variables are specified in the deployment descriptor - ejb-jar.xml for EJBs or web.xml for web applications - as shown below:

<env-entry>
  <description>Support Email Address</description>
  <env-entry-name>support.email</env-entry-name>
  <env-entry-type>java.lang.String</env-entry-type>
  <env-entry-value>support@company.com</env-entry-value>
</env-entry>

The problem is that the deployment descriptor also contains important configuration information that does not vary per environment. Worse, the deployment descriptor file is bundled into the EJB jar file or WAR file for deployment. So how exactly can you specify different values for environment variables, without complicating your deployment process? You cannot.

The deployment descriptor represents a good idea but a flawed design. The creators of the Java EE specification envisioned multiple roles, including not just a developer role, but also an application assembler role and a deployer role. The idea was that the developer could specify the default value for the environment variable, and it could be overridden by either the assembler or deployer. However, the specification does not specify an easy way to do this override, and implies that the assembler or deployer would have to directly modify the deployment descriptor or use the proprietary administration interface of the application server as is necessary for configuring datasources and connection pools. The other flaw with this design is the idea of these separate roles. In practice, the developers are also the assemblers and often the deployers as well. When there are separate individuals doing the deployment, they typically know nothing about the application and could only override settings based on instructions provided by the developers.

The Java EE platform provides good support for data sources, but other types of environmental differences are really not supported well. I recommend not using the environment variable mechanism provided in Java EE, as it does not address the requirement of easily deployable software.

My next example involves Ruby on Rails, a web application framework for the Ruby language that has been receiving a lot of attention and hype in the last few years. One of the attractions of Rails is that it provides built-in support for many of the common features of web applications, including handling of environmental differences. Rails explicitly defines the notion of an environment via the RAILS_ENV environment variable, and even comes pre-configured with three: development, test, and production. Specifying environment-specific settings is extremely simple. There is a common file shared between environments (config/environment.rb), and one configuration file per environment (config/environments/<env>.rb). Since the configuration files are Ruby scripts, any type of setup can be done - you are not limited as in the case of Java EE deployment descriptor XML files. Rails is a clear winner over Java EE when it comes to deployability for multiple environments.

Parameterize and lookup differences by environment

If the platform you are using does not provide the support you need for multiple environments, then the next design approach is to implement your own support for multiple environments within the platform. While the specific mechanisms can vary, conceptually such support requires a parameter representing the environment (like the RAILS_ENV environment variable), and a lookup mechanism to retrieve environment-specific values based on this parameter (like the config/environments/*.rb files in Rails). The lookup mechanism can be as simple as settings in a property file named after the environment, or can involve properties in a database table keyed by the environment. A more sophisticated mechanism is to use a configuration interface with a factory to create the appropriate environment-specific subclass based on the environment. This allows for any type of setup to be implemented, rather than being limited to strings or other primitive types.

The catch with using this approach is that the parameter representing the environment must be handled using whatever support the platform provides for dealing with environmental differences. For a Java EE application this is not a big deal - there are several different options I have used. One approach is to store the environment in a special database table. Since Java EE handles the data source fine, the code can retrieve the data source and then query the environment table to determine the environment. Another approach is to use a Java system property or even an environment variable like Rails does. A third approach I have used is to have an environment specific directory (i.e. config/prod/) holding one or more property files or other resources. The classpath is defined within each application server to include the appropriate environment-specific directory. The code simply loads the property files or other resources from the classpath, which resolves correctly to the desired version of the files. This works especially well for log4j configuration files.

For platforms that provide absolutely no support for handling environmental differences, this design approach will not work. That is when the third approach becomes useful.

Generate per environment

This design approach is most appropriate when the platform provides no support for handling environmental differences. The most common example of this I have encountered is database DDL statements, which can have environment-specific storage settings but do not support variables that would allow one to parameterize these settings. If you want to fully script database changes, then it is necessary to have these DDL scripts be environmentally neutral. The solution is to use file generation to produce a different version of the script for each environment from a template using the appropriate values to substitute into the template for each environment.

I provide a software utility called EnvGen on my Software page that performs environment-specific file generation. EnvGen is an Ant task for generating different versions of the same file parameterized for different environments (i.e. development, test, and production). File generation is done using FreeMarker, a template engine with a full-featured templating language. You specify environment-specific properties in a CSV file (comma separated value spreadsheet). You can read more about EnvGen in the EnvGen Release Documentation.

For all of these approaches, no matter what language or platform you are using, the underlying concept remains the same: separate the settings and code that changes across environments from that which remains the same to achieve the goal of reliably and easily promoting the code into the production environment.