Integration Tests with Maven (Part 2): Test Coverage Reports

In the previous article, we demonstrated one way of running integration tests with Maven. In this article, we will demonstrate how to measure the test coverage of both integration and unit tests.

We will be using Emma to measure and report the coverage. EMMA is an open-source toolkit for measuring and reporting Java code coverage.

Unit test coverage

Let’s start by generating a coverage report for unit tests only. This will require 2 new plugins to be added to our pom. The first one:

37
38
39
40
41
42
43
44
45
46
47
48
49

<plugin>
  <groupId>org.sonatype.maven.plugin</groupId>
  <artifactId>emma-maven-plugin</artifactId>
  <version>1.1</version>
  <executions>
    <execution>
      <phase>process-classes</phase>
      <goals>
        <goal>instrument</goal>
      </goals>
    </execution>
  </executions>
</plugin>

This plugin will instrument classes at target/generated-class/emma/classes, this won’t change the original classes at target/classes. Then, we must tell surefire plugin to look for classes at new location:

51
52
53
54
55
56
57
58

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.4.3</version>
  <configuration>
    <classesDirectory>${project.build.directory}/generated-classes/emma/classes</classesDirectory>
  </configuration>
</plugin>

At this point, we do 50% of code coverage. Mainly because the psvm ( public static void main ) is never executed.

Here is the EMMA text report. Emma does produce xml and html reports in addition to this simple text report:

[EMMA v2.0.5312 report, generated Fri Jun 12 15:43:28 BRT 2009]
-------------------------------------------------------------------------------
OVERALL COVERAGE SUMMARY:
 
[class, %]	[method, %]	[block, %]	[line, %]	[name]
100% (1/1)	67%  (2/3)!	56%  (10/18)!	50%  (4/8)!	all classes

Note: at this point no report is generated, we will configure report generation after integration tests coverage reports are in place. Once you run Surefire, you may notice the coverage.ec in the project root directory. This file contains the coverage result in a binary format used by Emma.

Integration test coverage

So far, we have the unit tests results. As I demonstrated in the previous article in this series, the integration tests need the project artifact to run, so, we need to instrument this jar in order to get coverage result on ITs. Let’s see the pom:

60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

<plugin>
  <groupId>org.sonatype.maven.plugin</groupId>
  <artifactId>emma4it-maven-plugin</artifactId>
  <version>1.3</version>
  <executions>
    <execution>
      <id>instrument</id>
      <goals>
        <goal>instrument-project-artifact</goal>
      </goals>
      <configuration>
        <appendEmma>true</appendEmma>
      </configuration>
    </execution>
  </executions>
</plugin>

Note that is using a different plugin: emma4it-maven-plugin. This plugin provides some additional goals for instrumenting integrations tests. For this project we use instrument-project-artifact goal. This goal will add Emma instrumentation into the project jar. This is specific to tests running on integration-test phase, because it does requires that the packaged artifact be instrumented.

There is also the configuration appendEmma. When true, this goal will shade the Emma jar into the instrumented jar. This is very useful to test jars that will be launched from command line, like this project.

[EMMA v2.0.5312 report, generated Fri Jun 12 16:05:45 BRT 2009]
-------------------------------------------------------------------------------
OVERALL COVERAGE SUMMARY:
 
[class, %]	[method, %]	[block, %]	[line, %]	[name]
100% (1/1)	100% (3/3)	89%  (16/18)	88%  (7/8)	all classes

At this point we have a very decent code coverage, almost 90%, 7 of 8 lines of code, and on this project it is not possible to get 8/8. And why is that? System.exit(). when this line is invoked nothing else is executed, that includes Emma instrumented code, so Emma won’t be able to know if System.exit() was executed, our integration tests catch that, because we check the exit code of the process, but Emma can’t do the same.

Test Coverage Report

At this point we have all coverage data stored at coverage.ec, in Emma binary format. Let’s now convert that into something that is more readable.

Add the following execution on emma4it-maven-plugin:

74
75
76
77
78
79
80
81
82
83
84
85
86
87

<execution>
  <id>report</id>
  <phase>post-integration-test</phase>
  <goals>
    <goal>report</goal>
  </goals>
  <configuration>
    <sourceSets>
      <sourceSet>
        <directory>${project.build.sourceDirectory}</directory>
      </sourceSet>
    </sourceSets>
  </configuration>
</execution>

Now the text, HTML, and XML reports are generated at target/site/emma.

Emma HTML coverage report

That is it for part 2. On the next part, I will show some ideas about testing more complex projects like flexmojos and nexus.

How-To, Maven emma, Maven, testing