Many software developers do not pay enough attention to the shutdown (also referred to as undeploy) of the applications they write. In fact, the most popular shutdown mechanism is to execute Ctl+C or a kill -9. However, if you are writing an application that coexists with other applications on the same application server, then a kill of the application server kills all the resident applications. You may need the ability to shut down your application, ensure that all resources consumed by it are released, and clear its memory footprint while leaving the other deployed applications running. In this article, I provide code samples for how to do that, and identify a set of checks and techniques.

Besides releasing resources, another important aspect of the shutdown process is ensuring your system is in a consistent state and mission-critical data is not lost. If your application processes transactions via Web service calls or batch loads, it is important that you bring those ongoing transactions to a halt smoothly; this article examines some techniques that can help you achieve that goal. Further, I discuss the usage of tools like JConsole and Borland Optmizeit to ensure a clean shutdown.

I will desist from presenting a formal definition of Web application shutdown. But, for practical purposes, one can say that an application has shut down when all the threads either started by it or servicing it have terminated their processing, and when memory used by the application during its lifetime, including object instances and class definitions, becomes unreachable from any of the currently live threads. Effectively, the memory used by the application during its lifetime becomes a candidate for garbage collection.

Shutdown event propagation

Application server front ends are available that allow you to deploy and undeploy Web applications. In the case of JBoss and Tomcat, the shutdown front ends are based on Java Management Extensions (JMX): when your application is deployed, the application server creates a managed bean (MBean) to manage its lifecycle, and when you undeploy an application, that MBean's undeploy method is invoked. You can also invoke such MBean operations from command line utilities. In addition to the JBoss JMX Console, JBoss supplies a utility called twiddle that invokes MBean operations from the command line.

When you undeploy a Web application, the container (through MBeans, in the case of JBoss) ultimately calls the destroy method on all active servlets. Applications typically have a singleton launcher servlet that performs the initialization and is placed as the first servlet definition in the web.xml file. Servlets are initialized in the order in which they are defined in web.xml and destroyed in reverse order. Cleanup code can be written in the launcher servlet's destroy method.

In standalone applications, you can notify your code to initiate an orderly shutdown through JVM hooks.

In this article, I take the case of a simple Web application that helps a database operations administrator request for, analyze, or truncate tables via a Web UI, and illustrate how shutdown is implemented. The operations requested via this application can take a long time to complete and are handled asynchronously by a thread pool, as illustrated in the code below:

 

public class LauncherServlet extends HttpServlet { static final String COMMAND = "command"; static final String TABLE_NAME = "table_name"; static final String ANALYZE_TABLES = "analyze_table"; static final String TRUNCATE_TEMP_TABLE = "truncate_temp_table"; private ThreadPoolManager m_threadPoolManager = null; private List<Statement> m_statementRegistry = null; private List<Connection> m_connectionRegistry = null; public enum ApplicationState { Unitialized, Running, ShutdownRequested, Shutdown }; static ApplicationState appState = ApplicationState.Unitialized; public LauncherServlet() { } /** * Container calls init during servlet load. */ public void init(ServletConfig servletConfig) throws ServletException { super.init(servletConfig); //Initialize your Thread pool. m_threadPoolManager = new ThreadPoolManager(); //Application initialization code. this.m_statementRegistry = new Vector<Statement>(); this.m_connectionRegistry = new Vector<Connection>(); setAppState(ApplicationState.Running); } /**

* Get */ public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { String command = request.getParameter(COMMAND); String table = request.getParameter(TABLE_NAME); String query = null; if (command != null && command.equals(ANALYZE_TABLES) ) { query = "ANALYZE TABLE " + table + " COMPUTE STATISTICS"; } else if (command != null && command.equals(TRUNCATE_TEMP_TABLE)) { query = "TRUNCATE TABLE " + table; } else { throw new ServletException("Command:" + command + " not recognized"); } SQLExecutor sqlExec = new SQLExecutor(this.m_connectionRegistry, this.m_statementRegistry, query); m_threadPoolManager.submitTaskForExecution(sqlExec); //In a real world application, an ID would be passed back to the client to track the status of the request. response.getOutputStream().println("<html><BODY><B> Request Submitted for Execution <B> <BODY></HTML>"); response.getOutputStream().flush(); } /** * Post */ public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { doGet(request,response); } /** * Container calls destroy at the end of the Servlet lifecycle. */ public void destroy() { //Set the application state. setAppState(ApplicationState.ShutdownRequested); //First shut down the thread pool, this call will attempt to //kill all the worker threads that are performing tasks. Be aware that //there is no guarantee that all the threads will die after this method call. try { m_threadPoolManager.shutdown(); } catch (InterruptedException e) { //The current thread that is calling destroy itself //may be interrupted by the container. In this case, //we want to do as much cleanup as possible so we will consume //this exception and not propagate it. System.out.println("Thread " + Thread.currentThread().getName() + " received interrupt, thread pool destroy was not complete"); } //Clean up all the resources. //Doing so, also increases the likelihood that any live threads at this point will die. //In our example, the resources are JDBC Connection and Statement objects. for (Statement stmt : this.m_statementRegistry) { try { //Statement.cancel will cancel the statement and will cause //the thread that is using this statement to get an SQLException. stmt.cancel(); stmt.close(); } catch (SQLException e) { ; //Move on and try to clean up other Statement objects. } } for (Connection conn : this.m_connectionRegistry) { try { //Good idea to rollback as some database systems will commit on close; //an attempt to commit might run into locking problems in some situations. conn.rollback(); conn.close(); } catch (SQLException e) { ; //Move on and try to clean up other Connection objects. } } //Unregister any components (MBeans for example) that you may have registered //with the container. setAppState(ApplicationState.Shutdown); System.out.println("Shutdown Complete"); }

public synchronized static ApplicationState getAppState() { return appState; }

public synchronized static void setAppState(ApplicationState appState) { LauncherServlet.appState = appState; } }

Shutdown sequence

The shutdown sequence can be implemented in two phases. In the first phase, set the application state to indicate that shutdown has been initiated and proceed to execute Thread.interrupt on all the threads your application has started (the subsequent section examines the details)—waiting a brief amount of time for the threads to die. However, you cannot be sure that they will die in the specified amount of time; thus, the next phase will help here.

In the second phase, clean up and undeploy all the resources that your application used, e.g., Java Database Connectivity (JDBC) connection pools, Java Message Service (JMS) session pools, or sockets. At this stage, any threads that are still alive and using resources should receive exceptions and die. Finally, unregister components that may have been registered with the container such as MBeans. Figure 1 illustrates the sequence diagram for shutdown.

Figure 1. Shutdown sequence diagram. Click on thumbnail to view full-sized image.

Thread termination

Thread termination is the first task to initiate as part of the shutdown sequence. This may also be the most challenging task especially if you are using a set of third-party tools that may spawn threads carelessly. Keep in mind that the Enterprise JavaBeans specification prohibits enterprise beans from starting threads; however, many non-EJB Web applications create threads that perform asynchronous processing of requests like the sample database administrator application. Instead of creating threads in your application, use a thread pool or a thread factory class that has a handle to all the threads your application has started. If you use J2SE 5.0 or later, use the java.util.concurrent.ThreadPoolExecutor as your thread pool. During shutdown, you can then call ThreadPoolExecutor.shutdownNow as shown in the following code listing:

 public class ThreadPoolManager {
  
  ThreadPoolExecutor m_threadExecutor;
  
  public ThreadPoolManager() {
//  ThreadPoolExecutor(int corePoolSize, 
//  int maximumPoolSize, 
//  long keepAliveTime, 
//  TimeUnit unit, 
//  BlockingQueue<Runnable> workQueue) 
//  You can also choose to provide your own ThreadFactory.
//  If ThreadFactory is not specified as in this case, 
//  a defaultThreadFactory will be created.
    this.m_threadExecutor = new ThreadPoolExecutor(2, 5, 5000, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<Runnable>(100));
  }
  
    /**
   * Submit a Task to the ThreadPool for execution.
   * @param task Runnable object that the ThreadPool will execute.
   */
  public void submitTaskForExecution(Runnable task) {
    m_threadExecutor.execute(task);
  }
  
    /**
   * Shut down the thread pool.
   * @return true if all the threads terminated at the time of return,
   *         else false.
   */
  public boolean shutdown() throws InterruptedException {
    //Alternative is shutdown(), which will wait for
    //all executing tasks to complete while stopping new tasks.
    m_threadExecutor.shutdownNow();
    //Put the current thread to sleep for a few seconds
    //to allow for all executing threads to complete.
    m_threadExecutor.awaitTermination(10000, TimeUnit.MILLISECONDS);
    return m_threadExecutor.isTerminated();
  }
}

If you are using a pre-5.0 JDK, you can use the EDU.oswego.cs.dl.util.concurrent.PooledExecutor class from the open source concurrent processing utilities library concurrent.jar from Doug Lea (the predecessor to the java.util.concurrent package).

Note that a thread cannot be stopped or destroyed on demand. You may only interrupt the thread and write the associated Runnable object's run() method or the subclassed Thread object's run() method in such a way that frequent checks verify whether an interrupt has been set and that any InterruptedExceptions are not ignored. This is true no matter which approach you use.

Some notes that will be helpful in writing your run() method logic:

  • Never consume the InterruptedException in your run() method. If a shutdown has been requested, always propagate it.
  • If you call interrupt() on a thread, it will run into an InterruptedException only if it is in a wait or sleep state. If the thread is active, it will not receive an exception; nevertheless, the interrupt flag will be set on the thread, which can be checked using Thread.isInterrupted().
  • If you have threads that process data for long periods of time or threads that are in an eternal loop, you should perform frequent checks to see if the interrupt flag has been set.
  • Not all interrupts mean shutdown; sometimes threads can receive interrupt signals due to other reasons. Therefore, you must check whether the interrupt was the result of a shutdown before you make the thread exit the run() method. Perform this check by looking at the application state.
  • Further, if a thread has not yet started, but has just been constructed, calling an interrupt on it will not set the interrupted flag. You can work around this case by checking for the application state up-front in your run() method.

The following code listing shows the implementation of a simple Runnable object that performs database queries and handles interrupts and exceptions properly to ensure a smooth shutdown:

Page 2 of 2
 public class SQLExecutor implements Runnable {
  List<Connection> m_connRegistry = null;
  List<Statement> m_stmtRegistry = null;
  String m_query = null;
  
  public SQLExecutor(List<Connection> connRegistry,
      List<Statement> stmtRegistry,
      String query) {
    m_connRegistry = connRegistry;
    m_stmtRegistry = stmtRegistry;
    m_query = query;
  }
  
  public void run() {
    String dsName = "java:samplesdb";    
    DataSource ds = null;
    Connection conn = null;
    PreparedStatement pstmt = null;
    boolean fCommit = false;
    
    if (LauncherServlet.getAppState() == LauncherServlet.ApplicationState.ShutdownRequested
        ||
        LauncherServlet.getAppState() == LauncherServlet.ApplicationState.Shutdown) {
      //Terminate this thread right away.
      return;
    }
    
    try {
      ds = (DataSource) (new InitialContext()).lookup(dsName);
    } catch (NamingException e) {
      e.printStackTrace();
      return;
    }
    try {
      conn = ds.getConnection();
      conn.setAutoCommit(false);
      m_connRegistry.add(conn);  
      pstmt = conn.prepareStatement(m_query);
      m_stmtRegistry.add(pstmt);
      //A good point to check if an interrupt has occurred.
      if (Thread.currentThread().interrupted() && 
          (LauncherServlet.getAppState() == LauncherServlet.ApplicationState.ShutdownRequested
              ||
              LauncherServlet.getAppState() == LauncherServlet.ApplicationState.Shutdown)) {
        throw new InterruptedException("Shutdown Signal Received");
      }
      pstmt.execute();
      conn.commit();
      fCommit = true;    
      System.out.println("Succesfully executed query: " + m_query);
    } catch (Throwable e) {
      //Make sure to always propagate exceptions to ensure smooth shutdown.
      //If you must handle exceptions without propagating them, exercise caution.
      throw new RuntimeException(e);
    } finally {
      if (pstmt != null) {
        try { 
          pstmt.close(); 
        } catch (SQLException ex) { 
          //Log it, move on, and clean connection. This may not be a critical.            
        }
        m_stmtRegistry.remove(pstmt);
      }
      if (conn != null) {
        if (!fCommit)  {
          try { 
            conn.rollback();
          } catch (SQLException ex) {
            //Log exception, send an alert to any monitoring software that
            //you may be using. Your DB server or network may be down.
          }
        }
        try {
          conn.close();
          m_connRegistry.remove(conn);
        } catch (SQLException ex) {
          //Log exception, send an alert to any monitoring software that
          //you may be using. Connection.close also returns the connection back to its pool,
          //and if this method fails, you potentially risk leaking a resource.
          throw new RuntimeException(ex);
        }
      }
    }
    
  }
  
}

Blocking I/O

Threads may be in listen mode waiting for data on a TCP/UDP socket. These threads are generally not interruptable because the underlying I/O is blocking (when the thread encounters a blocking I/O request, it is in neither the wait nor sleep state and therefore cannot be interrupted). This kind of situation happens when, for example, you are reading a chunk of data from a file and the thread executing the read may be blocked on I/O. A solution here is to use java.nio package support for InterruptibleChannel (available since J2SE 1.4) instead of the standard java.io classes that are not interruptible for your read/write operations. An InterruptibleChannel can be asynchronously closed by another thread. In our case, the shutdown thread can be made to close such channels. Several implementations of the InterruptibleChannel are available, including FileChannel and SocketChannel.

Request processors

Request-processor threads are the container threads that service the UI requests. At shutdown, several UI request-processor threads could be active. Such threads are out of your application's control, and the container should handle any issues with resources being held by those threads and their termination/reallocation.

You must be particularly careful with thread local variables in this context. There is no easy way to access ThreadLocal variables given a Thread object, as these variables are held in a WeakHashMap in the java.lang.Thread, which has package access only. However, since the ThreadLocal object is held in a WeakHashMap, ultimately the weak reference will disappear when your ThreadLocal object has no other strong references.

Third-party libraries

If a thread is executing code within a third-party library you are using and if that third-party code has not been written to react to interrupts or lacks a destroy/undeploy method, you may have to use other indirect techniques to effectively interrupt the thread. Also, some third-party libraries can spawn threads. An indirect way to terminate these threads is to undeploy the resources they may be using.

Resource undeployment

In the second phase of the suggested shutdown sequence, the application should undeploy/cleanup/release all the resources it typically uses—such as JDBC connections, JDBC statements, sockets, or JMS objects. The shutdown specifics for each resource type vary. In the simplest case, it may be a matter of iterating through the currently utilized resources and calling close/cleanup type operations on them.

A common scenario with enterprise applications is when several threads try to update the same data. If the transaction isolation level is set to read-committed, then only one thread's query will obtain row locks and others will be in lock-wait mode at the database. Executing an interrupt on the thread in this situation may not lead the thread to receive an exception because the underlying JDBC implementation may be blocked on I/O. To solve this problem, during the resource undeployment phase of the shutdown, traverse through each Statement that your application is using and execute a Statement.cancel on it. Doing so should cause the thread that is blocked on executeUpdate() or other such call to run into a SQLException and die (if the exception is properly propagated).

This type of resource management burdens your application, and you must check to see if your container provides advanced features for such cleanup tasks. For example, JBoss provides MBeans for the resources it deploys, such as JDBC connection pools, and stop/start operations are exposed through those MBeans.

Application semantics

In this section, we examine techniques to design your application in such a way that it does not lose critical data during shutdown. In large scale transactional systems and bulk data transfer systems, an orderly shutdown must ensure that transactional state and data is not compromised and no data is lost. For example, a transform- and load-type operation may be happening at the time of the shutdown request; in this case, a large file may have been partially transformed and if an abrupt shutdown occurs, the file may be corrupted. One technique that I have found useful is to have the processing code (in your Runnable) organized as a pipeline of tasks. Each task is atomic and, once completed, will write its state to a task log table. In addition, the data processed from the last successful task is always persisted. During startup, a check is made to the persistent store to see if any incomplete loads were in the pipeline at the time of shutdown. If an incomplete load is found, fetch the persisted state and associated data of that load's last successful task, and pass that to the next task in the pipeline.

When dealing with external systems that provide data through Web services or RMI-like (Remote Method Invocation) interfaces, as a part of your protocol, you must ensure that incoming data is persisted. In Figure 2, a simple data processing pipeline is shown with three stages. The first interfaces with the system delivering the payload and stores the data contained in the payload. The second stage transforms the data, and the final stage loads the data into some target database (not shown). At the end of each stage, the outputs are serialized and stored, and a task log details the completion of that stage.

Once your processing code is structured this way, you do not have to wait until the completion of all tasks in the pipeline. As soon as the current task is complete, a check can be made to see if the thread received an interrupt due to shutdown; if it did, make the thread exit the run() method.

Figure 2. Data processing pipeline. Click on thumbnail to view full-sized image.

Tools

The key to ensuring a successful shutdown implementation is to use proper tools to verify that your application is not leaving its footprints upon exit. Tools like Borland Optimizeit and YourKit Java Profiler can help you by letting you inspect the heap and the paths from garbage collection roots as well as list active threads. JConsole is a tool that ships with J2SE 5.0 and can show you the list of active threads and their stacks as well as memory usages and count of classes being used. However, it doesn't support a complete analysis of the Java heap.

When analyzing the heap, a good check is the absence of a Web application classloader instance associated with your Web application after a clean shutdown has occurred. All the objects (therefore threads) that your application created during its lifetime will have a reference to your application's Web application classloader. If the classloader associated with your Web application is no longer on the heap, you can rest assured that your application has left the building clean. When using Borland Optimizeit, you can filter the classes on the heap and just analyze the Web application classloader instances; in the case of JBoss 4.x, the Web application classloader name will be org.jboss.web.tomcat.tc5.WebAppClassLoader. You can find out your Web application classloader class name by writing a small piece of code in a JavaServer Pages (JSP) page as follows:

 <%@ page import="samples.LauncherServlet" %>
<html>
<head><title>Gets Web App Class Loader Name</title></head>
<body>
<P>
<B> Web App Class Loader Name is <B>
<P>
<%
out.println(samples.LauncherServlet.class.getClassLoader().getClass().getName());
%>
</body>
</html>

To run JConsole, add the following JAVA_OPTS to your startup script:

 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -
   Dcom.sun.management.jmxremote.authenticate=false -
   Dcom.sun.management.jmxremote.port=[chose a port number]

To launch the JConsole client, just execute jconsole from the command prompt after making sure that the JAVA_HOME and PATH variables are set properly.

Figure 3 shows the JConsole analysis of threads of the sample DB Operations Helper application while it is still deployed. After shutdown completes, the count of loaded classes falls back to pre-deployment levels and the threads associated with the ThreadPool disappear.

Figure 3. JConsole screen shot showing the list of active threads. Click on thumbnail to view full-sized image.

Clustering and failover

Most of the complexity in undeploying a clustered application should be handled by the container. Some applications, such as data transfer applications, may not be clustered, but will have failover implemented, meaning that at any given time, only one server is actively processing data, but a backup server is ready to spring to life should the primary fail. In such cases, a shutdown request should be propagated to the secondary before the primary. One way of coordinating this propagation is if the primary receives a shutdown request first (if the secondary is still exchanging heart beats), it could look up the MBean that manages the lifecycle of the secondary and in turn invoke a shutdown operation on that MBean before initiating its own shutdown. If the secondary receives the shutdown first, it should complete its own shutdown and then finally call an asynchronous shutdown on the primary.

Alternatives

Your application may be too complicated to analyze and adapt for a clean shutdown, or you may be resource constrained. In such cases, if you still must coexist with other applications in the same application server system, you can try horizontal-scalability-based configurations, where each Web application runs in its own JVM but is managed by an administrative server. That way, when the application is undeployed, the JVM it's running on will terminate.

Conclusion

It is best to design applications ground-up with the notion that they need to be brought down in an orderly manner without consuming system resources after they are shut down. The results of an abrupt termination of your application could be loss of data, resource leaks, threads/processes that continue to execute despite your shutdown request, and inconsistencies in your datastore. The approaches and tools discussed in this article should help you design, implement, and test a reliable, orderly shutdown mechanism for your Web application.

Ramkartik Mulukutla is a software architect at a supply-chain software provider. He has more than nine years of experience in architecting and developing enterprise Web-based applications, data warehouses, supply-chain software, middleware components, and has patented work in the area. He holds bachelors and masters degrees in computer science.

Learn more about this topic