Recently I conducted a load test on my beloved low end Ubuntu 8.04 VPS. The setup consists of a mysql 5.1 DB, a Tomcat 6 webapp, openjdk 6 and Varnish 2.1.2. The load generator was issuing HTTP GET requests through a DSL connection which were supposed to create a decent load on Tomcat.
To my surprise I found that after a short warmup phase, Tomcat wouldn’t answer any further requests because all requests would be served by Varnish at 300 requests / second with a negligible 3% CPU load. This could have been the happy end of a load test, but a brief visit to the webapp over a browser revealed that it would not serve web requests or if, at a very slow rate and for the most part just time out the connections.
Looking through both Tomcat and Varnish logs didn’t yield anything interesting, in particular no messages about errors or dropped connection. I also was able to discarded the suspicion that the OS was rejecting connections, as neither the load generator reported any dropped connections nor did any attempts to connect at SSH pose any particular problems.
It turned out that for the Varnish version I was using, an experimental feature called “session_linger” was getting in the way – and I wasn’t the first one to bump into it [1]. Fortunately the solution is just as simple, namely by reducing the value to 20 (i.e. in /etc/default/varnish)
[1] http://www.mail-archive.com/varnish-misc@projects.linpro.no/msg03071.html
George's Techblog 