Based on my experience, I thought I had a good appreciation for what can go wrong. That changed recently when I experienced a day filled with too many problems and errors to be believed. The day started off innocently enough until a member of the production support team came by to inform us that he had accidentally terminated the database connection of one of our batch jobs due to transposing two numbers in the identifier. This by fluke matched our job instead of the one he wanted. Okay, no problem, we simply need to confirm that nothing was corrupted and restart the process. We checked our email for the notification email that is sent when a batch job abnormally terminates in order to verify which job had been affected. No such email was found. A little puzzled, we checked the server and confirmed that the process was no longer running. But another batch process was executing, and we identified it as a subsequent batch job dependent on the first. Subsequent jobs only run if the predecessors execute successfully, so we had a sinking feeling as we started checking the log files. Sure enough, due to a complete lack of error handling, the first job had reported a successful execution despite the database connection failure, which had caused the second job to start. That explained the lack of a notification email. The second job depended on the processing results of the first job, so the output of the second job was suspect and likely wrong. We had to kill the second job. If the first job had just failed, we could have restarted it without a problem, but now we had to investigate how to undo the effects of both jobs and manually restart the first.

Well, that didn't seem too bad, until I had time to think for a second. That is when I realized that our batch jobs are always scheduled at night or the weekends, and never during business hours. What was one doing running during the day? That prompted another investigation, which revealed that the job does normally runs on weekends. But the previous weekend there were problems with predecessors to this job that caused it to be delayed until one of the nights during the week. So why didn't it run at night? We were surprised to discover that it had – it just didn't finish. Due to performance issues, the job had run for over eight hours, extending into the day, before it was killed by mistake.

By late afternoon of that day we had multiple investigations underway trying to track down the various root causes of the problems we had identified. My mind reached the saturation point sometime in the afternoon, so I cannot remember the details concerning what was found. I suspect there were other problems unearthed that I have since forgotten. Nor where we able to get everything fixed that same day. That combination of problems, coming together on that one day, kept a surprising number of people busy for days sorting out the mess.

I was upgrading a Java web application to WebSphere server version 6.1 and as the first step I switched the development environment to Rational Application Developer version 7. With the new IDE came an improved compiler that reported additional warnings, so it didn't surprise me to see hundreds of new warnings. It is a standard practice of mine to eliminate warnings, even harmless ones, since a significant warning can easily be missed if harmless warnings are allowed to remain. As I worked my way through the warnings, I came across a new one I did not recognize: "Finally block does not complete normally". A simplified version of the code producing this warning is shown below:

  boolean performBusinessOperation() {
    boolean operationResult = false;
    try {
      // Perform some business logic...
      operationResult = true;
    } catch (IllegalStateException e) {
      // Handle this exception..
      operationResult = false;
    } catch (IllegalArgumentException e) {
      // Handle this exception...
      operationResult = false;
    } finally {
      // Common cleanup...
      // Following line produces warning
      // "Finally block does not complete normally"
      return operationResult;
    }
  }

This was not code I had written, so I spent some time trying to figure out the reason for the warning. In Java, the finally block is guaranteed to be executed after the contained try block finishes execution, even if an exception is thrown within the try block. If an exception is thrown and not caught within a function, the exception is propagated up the call stack until it encounters an appropriate catch block. But in this particular case within the performBusinessOperation method, if an uncaught exception is thrown, the finally block will run and perform the return statement. So which will win - the exception or the return? I was not sure, which in my mind explained the warning - it is bad practice to write code with unclear behavior. So I fixed the code by moving the return statement outside the finally block and moved on to the next warning.

Once I was finished eliminating warnings, I ran the entire suite of automated unit tests. To my surprise, I had a few failures. When I tracked down the offending code, I was surprised to see that it was my fix for the "Finally block does not complete normally" warning that broke the tests. How could that be? After further tracing and debugging, I finally found the reason: the unit test incorrectly invoked the method in question, causing it to throw a NullPointerException from within the try block. Having the return statement within the finally block apparently was causing the exception to be silently discarded. I found this shocking. This is dangerous behavior for a language, and I had problems believing that was actually the case. So I wrote a quick unit test for verification, shown below.

public class ReturnInFinallyBlockExample
  extends junit.framework.TestCase
{
  @SuppressWarnings("finally")
  private boolean isReturnWithinFinally() {
    try {
      if (true) throw new RuntimeException();
    } finally {
      return true; // This hides the exception
    }
  }

  private boolean isReturnOutsideFinally() {
    try {
      if (true) throw new RuntimeException();
    } finally {
      // Return outside finally block.
    }
    return true;
  }

  public void testReturnFromFinallyBlockWithUnhandledException() {
    assertTrue(isReturnWithinFinally());
    try {
      isReturnOutsideFinally();
      fail("Expect exception");
    } catch (RuntimeException e) {
      // Expected case.
    }
  }
}

This test case passes, demonstrating that the uncaught exception within the try block is silently discarded if the return statement is within the finally block. Note my use of the Java 5 annotation @SuppressWarnings("finally") in order to stop the compiler from reporting the "Finally block does not complete normally" warning for this example.

Perhaps I should have been less surprised by this behavior in Java given that I was already aware of another suboptimal situation regarding Java exception handling in finally blocks: if an uncaught exception is thrown in a try block and then another exception is thrown in the finally block, it will be the second exception that is propagated out of the method. The first exception will be silently lost. The following test case demonstrates this behavior:

public class ExceptionInFinallyBlockExample
  extends junit.framework.TestCase
{
  private void haveExceptionInFinallyBlock() {
    try {
      if (true) throw new IllegalArgumentException();
    } finally {
      if (true) throw new NullPointerException();
    }
  }
  public void testHaveExceptionInFinallyBlock() {
    try {
      haveExceptionInFinallyBlock();
      fail("Expect exception");
    } catch (NullPointerException e) {
      // Expected case.
    }
  }
}

Ignoring errors is dangerous, as I have discussed in my article on Error Handling and Reliability. So I strongly feel that the Java language should have prohibited return statements in finally blocks. Fortunately, modern Java IDEs like Eclipse can make up for this shortcoming by allowing you to flag this code construct as an error rather than a warning.

The source code listed in this article is provided in the Java Examples project which can be downloaded from the Software page.

• Ignore errors
• Fail fast
• Degrade gracefully

The ignore errors approach is very simple: assume errors will not happen and ignore them. Some of you may object that this is not a 'real' error handling strategy, but considering how often I have seen it used in production systems, I cannot agree. This approach does have the benefit of maximizing uptime: even if things go wrong, the program will keep running. Of course, if the program is producing incorrect output due to these errors, then you have a problem. So this approach tends to minimize correctness. Any problems that do occur are what I call silent failures that go undetected, at least for a while. Unix scripts and the C programming language adopt this strategy as the default: utilities and functions have return codes to report errors, so a call that results in an error will not affect the operation of your program or script unless you have an explicit check.

The fail fast approach is also very simple: whenever an error or unexpected event happens, immediately terminate execution. This approach tends to maximize correctness, but tends to minimizes uptime, since any abnormality causes it to end. These applications tend to be brittle. The slightest problem in the environment, such as a blip in the network, can bring down the application. Modern enterprise programming languages such as Java and C# adopt this strategy through the use of exceptions. If a problem occurs, an exception is thrown which will terminate the program unless explicitly caught and dealt with.

The degrade gracefully approach combines the best of the other two approaches. It detects errors like the fail fast approach, but instead of failing immediately, it handles the error and continues execution as appropriate. It therefore maximizes the reliability of the system by maximizing both correctness and uptime. The downside of this approach is that it requires much more thought and effort to implement. No programming language I am aware of provides explicit support for this approach.

I was originally a strong proponent of the fail fast approach, but last year I started to appreciate the degrade gracefully approach, as I wrote in my article Fail Fast or Degrade Gracefully?. Over the past year, my viewpoint has shifted further. I now feel that the degrade gracefully approach should be used by default. Only if it would require too much effort or complexity to implement should the fail fast approach be used instead. (Naturally I do not support the use of the ignore errors approach.)

There are many examples of the degrade gracefully approach within the IT infrastructure we rely on. TCP/IP networking stacks are designed to degrade gracefully when problems such as dropped packets occur. Web servers do not shut down if a web application experiences a failure - they instead terminate the current request by sending an error response to the client and continue to serve other requests. Email clients do not fail if the email server becomes unavailable, and more importantly the mail you were trying to send is not lost. Modern compilers do not stop upon encountering the first syntax error but instead continue parsing the same file (and other files) as best they can.

The validity of these examples could be debated. One could argue that some of these situations such as dropped network packets and bad user input (syntax errors in code) are expected - a normal part of operation - rather than representing an exceptional situation or error. The systems handle these situations because it is a requirement, not because they are using the degrade gracefully error handling approach. I would instead argue that the requirement is to use the degrade gracefully approach to handle these problematic situations, primarily because both the ignore errors approach and the fail fast approach are unacceptable.

Reliable systems do not happen by accident, but require careful thought and effort to create. The approach you choose for handling errors can have a bigger impact on reliability than you might expect.

My most significant growth was in the area of personal productivity: I read and implemented the organizational system described in the book Getting Things Done : The Art Of Stress-Free Productivity by David Allen. I have found this system very useful both at work and at home, and wrote an article describing my experience implementing the system.

This last year I switched to a four day work week, and have found it to be a significant improvement over the normal five days. I even figured out how to do this without a drop in pay.

One area of software development I have explored a lot this year has been change and release management: specifically the process of promoting application or infrastructure changes into a production environment. Both my experiences at work and my experiences running this website have made me appreciate the necessity of having a defined process. The amount of process that is necessary depends on the types of changes being made and the complexity of the environment. As I wrote in my article on deploying application changes, it is easier when the application is packaged and deployed as a single unit. When this is not possible - like for database changes - then more process is necessary. Adding too much process, however, can be just as harmful as having too little - a proper balance must be maintained.

Over this last year I have spent a lot of time thinking about application reliability. Ensuring a system is highly reliable is a surprisingly difficult task, and one of the major culprits is complexity. I have found myself arguing more strongly for simpler, more reliable solutions as a result. My most recent focus has been on error handling. My article on the fail fast and degrade gracefully approaches to error handling contains some earlier thoughts on this subject, but I have not yet written up my latest ideas. I have come to believe that the degrade gracefully approach, while more difficult to implement, is the best for creating highly reliable software. I have unfortunately encountered a third approach to error handling: ignore errors. This leads to silent, undetected failures whose negative effects go unnoticed for an arbitrary length of time.

I have also learned a number of lessons from running this website, including improving my knowledge of web design, promotion, and on-line advertising. I have found the process of writing my weekly articles to be a great learning aid. Writing forces me to reflect on and clarify my thoughts and ideas on a particular subject. It is no coincidence that I have provided a number of links above to articles I have written this past year. I tend to write about topics that are currently in my focus of attention, and the act of writing about them helps clarify and solidify the learning I have done.

For quite a while I have been a firm proponent of the fail fast approach. If you encounter an internal application error (i.e. a method parameter is unexpectedly null), this is often a sign of a defect. The presence of a defect means you can no longer trust the operation of the application, so the safest approach is to terminate the operation or even the application. (In Java, this is typically done by throwing an appropriate RuntimeException.) Besides being the safer of the two approaches, another advantage of fail fast is that it forces the problem into the open, which makes it more likely it will be detected and fixed.

However, I recently came across a situation in which the degrade gracefully approach made more sense. The application in question had a generic message class for formatting messages with parametrized arguments. To use the class, you provide the message embedded with tokens representing one or more parameters, plus the parameters to be substituted for the tokens. One day I came across a use of this message class that supplied a parametrized message with the wrong number of parameters. Curious as to why this block of code had not 'died' (thrown an exception) during testing, I looked into the implementation of this message class. I discovered that the class did absolutely no checking of the arguments supplied to it. As a result, you could supply the wrong number of parameters (too many or too few), and the class would still return the formatted string, ignoring extra parameters and treating missing parameters as empty strings. A little investigation quickly revealed that there were other places in the application that were supplying the wrong number of parameters to this class.

So I refactored the message class to use the fail fast approach, then searched for usages of the class to fix the cases where the arguments were invalid. It didn't take that long before the changes were done and all the unit tests were successful, so I committed my code. Some time later someone encountered an error which I quickly recognized - an invalid argument supplied to that generic message class. Obviously, I had missed a place in the application that was calling the message class incorrectly. But the error had me think: the message class was used to format a message to be displayed to the user. Before, with the degrade gracefully approach, the users had been able to perform the operation in question successfully, despite getting a poorly constructed message. Now with the fail fast approach, we did quickly find out about the bad message, but the user could no longer complete the work they were trying to do. I wasn't happy about my change having made the application less useful for the user.

After some thought, I realized that the degrade gracefully approach was appropriate in this situation. A message to the user missing some parameters is almost always still somewhat understandable, and has nothing to do with the actual business logic being performed, so it is fairly safe to continue with constructing the message despite receiving the incorrect number of parameters. But I still wanted to be able to find out about these cases - they did represent defects (albeit minor) in the code. I really wanted the advantages from both approaches.

To achieve this, I again refactored the message class to allow it to proceed despite having the wrong number of parameters. I changed the code checking for invalid parameters to log an error to the application log instead of throwing an exception. By logging an error I ensured that the developers would find out about the problem, but the application would proceed. (You may be thinking that this error in the log file is likely to be overlooked by developers, but I had already implemented changes to ensure this wouldn't happen. I'll save the details of this for a future article.)

In most cases, I still prefer the fail fast approach. Even in this case involving the message class, if the original developers had used the fail fast approach then I suspect there would have been far fewer cases of calling code supplying the wrong number of parameters. This is a potential drawback of the degrade gracefully approach: if you are not careful, you end up hiding information about a defect. If you do decide to use the degrade gracefully approach, ensure you have a mechanism to detect and reveal any defects, rather than completely hiding them. One case where the degrade gracefully approach is often used is at the application architecture level. Applications such as web servers and business web applications that process multiple independent operations do not terminate upon encountering an internal error. Instead, the current operation is terminated with the appropriate error reported while the application continues running, able to process other requests.