Getting test coverage reports for integration test

I’m a vociferous supporter of integration tests with an embedded servlet container and in this post I’ll show how to get test coverage for the classes touched in those integration tests.

All test coverage gauging tools I know instrument code under test either at build time or run time through Java agents. Because the code under test runs in the servlet container’s class loader, it may either not be properly instrumented or evade recording of its execution.

The demo project [4] on Github consists of a multi-module Maven project: a rather (but not too) simple Java web application as a sub module and an integration test sub module. The integration test sub module is meant to start a Tomcat container with the web application, run HTTP integration tests on it and record test coverage.

Jacoco is a powerful test coverage recording framework which comes with a handy Maven plugin; check the project’s pom.xml for all the gory usage details. The integration tests with report generation are run like this:

mvn clean install

Test coverage reports can then be found in webapp-integration-tests/target/site/jacoco-aggregate.

The interesting parts:

0. Generating test coverage in a multi-module project isn’t trivial. The tests run in a different module that the code under test, which is then considered an external dependency. There is a good piece [5] on the topic and an instructive explanation [6] on Github.

1. The web application project is largely unaffected by the existence of integration tests; there are no special instrumentation or preparation steps in the pom of that project, which is non-invasive and hence good. “Largely unaffected” because of one minor concession: although a WAR project, the integration test module needs its classes as dependencies which it normally cannot get from the WAR dependency. This is easily fixed in pom.xml:

 <plugin>
  <artifactId>maven-war-plugin</artifactId>
  <version>2.2</version>
  <configuration>
   <attachClasses>true</attachClasses>
  </configuration>
 </plugin>

2. Much of the magic happens in the integration test module. I extensively have written about how to start Tomcat from a unit test, here I’m going to talk about the Maven setup.

The pom.xml contains three interesting plugin configurations:

(1) instructing Surefire to skip the test phase and run during the integration-test phase

(2) passing extra JVM instructions (which Jacoco prepared) to Surefire so that the latter runs Jacoco’s runtime agents

 <plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
   <version>2.20</version>
   <configuration>
     <trimStackTrace>false</trimStackTrace>
     <skip>true</skip> (1)
   </configuration>
   <executions>
     <execution>
       <id>integration-tests</id>
       <phase>integration-test</phase>
       <goals>
         <goal>test</goal>
       </goals>
       <configuration>
         <skip>false</skip> (1)
         <includes>
           <include>**/integrationtests/*Test.java</include>
         </includes>
      <argLine>${jacoco.agent.itArgLine}</argLine> (2)
     </configuration>
   </execution>
 </executions>
 </plugin>

(3) Setting up Jacoco with a report aggregate configuration for the report generation phase

 ...
 <execution>
   <id>jacoco-report</id>
   <phase>post-integration-test</phase>
   <goals>
     <goal>report-aggregate</goal> (3)
   </goals>
   <configuration>
     <includes>
        <include>twet/**</include>
     </includes>
   </configuration>
 </execution>
 <execution>
   <id>prepare-it-test-agent</id>
   <phase>pre-integration-test</phase>
   <goals>
     <goal>prepare-agent</goal>
   </goals>
   <configuration>
    <append>true</append>
    <inclNoLocationClasses>true</inclNoLocationClasses>
    <includes>
     <include>twet/**</include>
    </includes>
    <excludes>
      <exclude>org/**</exclude>
      <exclude>com/**</exclude>
      <exclude>net/**</exclude>
     </excludes>
    <propertyName>jacoco.agent.itArgLine</propertyName> (2)
   </configuration>
 </execution>

There are more interesting things going on here: we are interested in coverage only for our project, so a proper class include instructs Jacoco to generate coverage reports only over our project’s classes. A proper exclusion list also helps keeping run times low. The “report-aggregate” goal tells Jacoco that the code to cover comes from a different project.

And last not least, importing classes as dependencies from a WAR projects requires a special dependency declaration:

 <dependency>
   <groupId>georgovassilis</groupId>
   <artifactId>webapp-implementation</artifactId>
   <classifier>classes</classifier>
 </dependency>