Tomcat, Spring and memory leaks when undeploying or redeploying an web application

In this post I’ll talk about a new kind of memory leak in Spring applications involving transaction management and initializing beans.

While developing a web application with Spring and Tomcat I frequently run out of heap space. The web is full of stories about this. There is Aspirin to suppress the symptoms [1] and there are wise people who know why this is happening [2].

The short (and scientifically not 100% rigorous) story is that the web application started either a thread or registered an object with a global registry which it then forgot to stop or unload.

Tomcat 7 added [2] not only detection for many of these cases, but in some instances it also can remedy the situation by forcefully unloading JDBC drivers and terminating rogue threads.

I argue that it is ok if the application server cleans up after an application – in a way, it is not different than garbage collection (which is now socially accepted). However the sad fact is that it doesn’t (yet) work all the time which make Aspirins [1] against the headache necessary and just postpone the unavoidable, namely stopping and restarting the JVM.

The JDBC driver is parked with the application server, any thread pools are being closed when the application shuts down, quartz schedulers are also being cleaned up… and yet there is an entry like this every time the application stops:

SEVERE: The web application [] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources] ... but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

@Component
public class SomeComponent{
  @Autowired
  SomeSpringDataJpaDao dao;
  
  @PostConstruct
  public void setup(){
    dao.doSomething();
  }
}

What had happened here? Spring was instantiating SomeComponent, autowiring a Spring-data-jpa DAO into it and then calling the setup method. This runs a query against JPA without a transaction. Anyone who has worked enough with databases knows this means trouble. In my defence, SomeComponent participates (almost) always in a transaction which is driven by a ServletRequest. Ironically the particular case runs with a timer and thus does not go through the web layer which would have spanned the transaction.

The solution is (almost) fortunately quite trivial: add an @Transactional annotation to either the component or the setup() method.

Note that Spring-data-jpa 1.4.3 seems to have fixed a related bug in LockModeRepositoryPostProcessor.invoke() which used to set the “Transactionl resources” thread local variable but never cleared it.

Update 2018.05.27

I’m using this Spring application event listener to clean up lingering C3P0 and MySql threads. This is no original research; I stole the bits and pieces from around the web a while ago but, unfortunately, don’t recall where from. I’m sorry 😦

public class DeregisterJdbcDrivers implements ApplicationListener {

  Logger logger = Logger.getLogger(DeregisterJdbcDrivers.class);
  void stopMySql() {
	try {
		logger.info("Shutting down MySQL cleanup thread");
			AbandonedConnectionCleanupThread.shutdown();
	} catch (Throwable t) {
	}
  }

  void unregisterJdbcDrivers() {
	Enumeration drivers = java.sql.DriverManager.getDrivers();
	while (drivers.hasMoreElements()) {
		java.sql.Driver driver = drivers.nextElement();
		logger.info("Unregistering JDBC driver " + driver);
		try {
			java.sql.DriverManager.deregisterDriver(driver);
		} catch (Throwable t) {
		}
	}
  }

  public void stopC3P0() {
	C3P0Registry.getNumPooledDataSources();
	for (PooledDataSource dataSource : (Collection) C3P0Registry.getPooledDataSources()) {
		try {
			dataSource.close();
		} catch (Exception e) {
			logger.error(e);
		}
	}
  }

  @Override
  public void onApplicationEvent(ContextClosedEvent event) {
	stopC3P0();
	stopMySql();
	unregisterJdbcDrivers();
	try {
		Thread.sleep(2000L);
	} catch (Exception e) {
	}
  }
}