In the third part of this series I will talk about how integration tests (and unit tests) are done on Flexmojos. If you haven’t been following along, you should read the first and second posts of this series.

Let’s start from the beginning, what is Flexmojos? Flexmojos is a maven plugin for Adobe Flex development. You can get acquainted with it here.

Some Background

When I started Flexmojos I just wanted a decent maven plugin to build my project. I never considered making it public or turning it into a open source project. At the time, I just wanted something for me and for me alone. As I continued to work on my own solution, I realized more people had problems building Flex projects using Maven. So I decided to share my project. This is how the Flexmojos project began, and once the project had some momentum we stared running into challenges that required rigorous integration testing.

After a few releases, Flexmojos started to present backward compatibility issues. Something that worked on 1.0-alpha-2 suddenly stopped working on 1.0-alpha-3. And this started to happen more often during new features inclusion. The obvious fix for that was to create a bunch of unit tests. But mojos aren’t exactly the easiest things to unit test. So I had to keep testing manually, at that point Maven and integration tests had too much complexity for my limited Maven API knowledge. So I stopped including new features and concentrated on getting Flexmojos stable. Eventually I managed to get 1.0 release out.

I saw how much time, releases and code I needed to get 1.0 stable and I wanted to get 2.0 stable w/o that much work. But how? Creating a good set of automated tests. But again, how to test mojos? At this point, there wasn’t much information about testing Maven plugins. Googling here and poking around there I found maven-verifier.

maven-verifier tests are run using JUnit or TestNG, and provide a simple class allowing you to launch Maven and assert on its log file and built artifacts. It also provides a ResourceExtractor, which extracts a Maven project from your src/test/resources directory into a temporary working directory where you can do tricky stuff with it.

That was perfect for my needs. Using maven-verifier I found an easy way to test Flexmojos. In fact, I found a very good way to test Flexmojos, however when I started to write more complex tests it proved to be not as easy as I expected. We’ll talk more about that later. Using maven verifier is very cool too because it allowed me to write integration tests using TestNG, so I could use asserts, inherit, annotations and all the stuff available for unit testing that I was used to using.

How are Flexmojos test organized

Flexmojos only has integration tests. Unit testing maven plugins is not easy and Flexmojos wasn’t designed in a particularly modular way. If it were, it would have more unit tests.

Since Flexmojos is a Maven plugin I needed it installed into a Maven repository to run the integration tests, for that reason Flexmojos has all integration tests grouped into a single project, called flexmojos-test-harness.

The flexmojos-test-harness creates a complete environment to run the integration tests: – Download and unpack Maven 2.0.9 from central; – Overwrite Maven settings.xml with one that contains extra repositories ( check it here ), with all required artifacts; – Install Flexmojos artifacts into a fake local repository; this allow us to – Instrument Flexmojos artifacts using Emma; – Add emma.jar to Maven libs directory (required to enable test coverage).

This project sets up everything necessary to run Flexmojos (install maven), it sets up Emma (instrumenting and add emma.jar) and makes sure the right version of the artifacts are used during tests (filling maven repo).

Motivation for Flexmojos Integration Testing

Q. Why download and unpack maven from central, instead of using the maven already present on machine? A. First and most important reason: Instrumented classes need emma.jar on classpath. If not present, it will throw ClassDefNotFoundExceptions. So I can tamper with this Maven instance created by Flexmojos adding emma.jar. And that is a good reason to ensure that Flexmojos can manage a separate Maven installation that is isolated from all of the variation of developers who might be testing Flexmojos. At any moment I can set another version to be sure that Flexmojos is still working.

Q. Why overwrite settings.xml? A. First of all, it was was necessary to add the Flexmojos repository, so that maven could download the flex compiler from the server. This modified settings.xml does include a reference to an isolated, local Maven repository, so that anyone running the integration tests will retrieve artifacts already available on current machine, this dramatically reduces the time required to run the integration tests. Also notice that this settings.xml blocks all snapshots from being downloaded, that is nice because it prevents instrumented Flexmojos being overwritten with non-instrumented versions from maven central.

Q. Why put the artifacts on a fake local repository? A. That is about isolation… if I used the machine’s local repository, Flexmojos artifacts would get instrumented by Emma. But that would break Flexmojos. It’s required to add emma.jar on Maven classpath just to run instrumented Flexmojos. But shipping instrumented Flexmojos gives users no advantage, quite the opposite, it would generate coverage reports every time, wasting CPU, disk and memory usage. Another reason is that would break the GPG signature for Flexmojos artifacts. So, creating a isolated local repository allowed me to run instrumented tests without messing with built artifacts.

How to Configure Integration Tests

Q. How do you download and unpack maven from central? A. To copy and unpack artifacts is used the maven-dependency-plugin. This plugin “provides the capability to manipulate artifacts. It can copy and/or unpack artifacts from local or remote repositories to a specified location.”

58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

<execution>
  <id>extract-maven</id>
  <phase>generate-test-resources</phase>
  <goals>
    <goal>unpack</goal>
  </goals>
  <configuration>
    <artifactItems>
      <artifactItem>
        <groupId>org.apache.maven</groupId>
        <artifactId>apache-maven</artifactId>
        <version>${maven.version}</version>
        <type>tar.gz</type>
        <classifier>bin</classifier>
      </artifactItem>
    </artifactItems>
    <outputDirectory>${project.build.directory}</outputDirectory>
  </configuration>
</execution>

Read more about unpack at plugin site. Unpack goal documentation and more examples.

Q. How does settings.xml know where to find the local Maven repository? A. The settings.xml used on tests has some variables, namely ${localRepository.basedir}, that links to default repository location, and ${maven.repo.local}, that has the location when default-value is overwritten. This variables are interpolated using maven-resource-plugin. To enabled the variables interpolation is only necessary to declare the resources with the filtering option set to true, like this:

43
44
45
46
47
48
49
50
51
52
53
54
55

<build>
  <testResources>
    <testResource>
      <directory>src/test/resources</directory>
      <filtering>true</filtering>
      <includes>
        <include>settings.xml</include>
      </includes>
      <targetPath>${fake-maven}/conf</targetPath>
    </testResource>
...
  </testResources>
</build>

That was not declared on maven-resource-plugin configuration, but on build section, under resources or testResources (Flexmojos ITs case).

Q. How does the resources directory get interpolated? And why isn’t maven-resource-plugin used? A. That is a performance question…. maven-resource-plugin would always interpolate all files, and there are several files under resources, that alone was taking 30 seconds to a minute. But when running individual tests, not all files are required, so Flexmojos implemented its own interpolation, which runs before the test. Take a look at AbstractFlexmojosTests.getProject()

Code coverage

For the last part of this blog, I show how Flexmojos generates a test coverage report. Flexmojos uses emma4it-maven-plugin (introduced on part 2 of this blog series).

Emma4it has a goal to merge all available Emma binaries into a single one.

76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

<plugin>
  <groupId>org.sonatype.maven.plugin</groupId>
  <artifactId>emma4it-maven-plugin</artifactId>
  <version>1.3</version>
  <executions>
    <execution>
      <id>merge</id>
      <phase>verify</phase>
      <goals>
        <goal>merge</goal>
      </goals>
      <configuration>
        <searchPath>${basedir}</searchPath>
      </configuration>
    </execution>
...
  </executions>
</plugin>

This way, it is possible to combine coverage results of unit and integration tests, it becomes more relevant on Flexmojos because we have being improving UT tests coverage, especially in some new components that are very easy to unit-test (like the new generation and testing API).

In part 4 we will talk a bit about ITs on Nexus. Nexus does have a very particular setup where the approach used for Flexmojos proved to be inefficient.

See you in the next post.

Everyone knows Maven is great for running unit tests, and it is usually one of the first things that people learn when they are adopting Maven as a technology. Integration tests are another matter, and require a more detailed introduction. In this series of articles, I will explain how to set up integration tests in Maven starting the series by testing a simple jar and then advancing into more complex scenarios.

The application

Let’s define a very simple application to be tested. The application will return an exit code equal to the number of command-line parameters. Running the program with no parameters will produce an exit code equal to zero, two parameters exit code two, and so on.

The application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

package org.sonatype.simpleclientapp;
 
public class Main
{
    public static void main( String[] args )
    {
        System.exit( execute( args ) );
    }
 
    public static int execute( String[] args )
    {
        if ( args == null )
        {
            return 0;
        }
 
        return args.length;
    }
}

This class is very simple to test, in the next section you will see how you can write a simple integration test using JUnit.

Creating and running integration tests

Let’s create a simple integration test using JUnit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

package org.sonatype.simpleclientapp;
 
import java.io.File;
 
import junit.framework.TestCase;
 
public class MainIT
    extends TestCase
{
 
    public void testExecute()
        throws Exception
    {
        assertEquals( 0, execute( new String[] {} ) );
        assertEquals( 1, execute( new String[] { "one" } ) );
        assertEquals( 6, execute( new String[] { "one", "two", "three", "four", "five", "six" } ) );
    }
 
    private int execute( String[] args )
        throws Exception
    {
        File jar = new File( "target/simple-client-app-1.0-SNAPSHOT.jar" );
 
        String[] execArgs = new String[args.length + 3];
        System.arraycopy( args, 0, execArgs, 3, args.length );
        execArgs[0] = "java";
        execArgs[1] = "-jar";
        execArgs[2] = jar.getCanonicalPath();
        Process p = Runtime.getRuntime().exec( execArgs );
        p.waitFor();
        return p.exitValue();
    }
 
}

Note that this test needs the packaged jar (simple-client-app-1.0-SNAPSHOT.jar), which isn’t created when surefire runs the unit tests.

To run this we need a different approach. By different approach I mean a different plugin: the Failsafe Maven Plugin. The Failsafe Plugin is a fork of the Surefire plugin designed to run integration tests.

The Failsafe plugin goals are designed to run after the package phase, on the integration-test phase.

The Maven lifecycle has four phases for running integration tests:

• pre-integration-test: on this phase we can start any required service or do any action, like starting a database, or starting a webserver, anything… you can think on this as Junit.setUp()
• integration-test: failsafe will run the test on this phase, so after all required services are started.
• post-integration-test: time to shutdown all services… you can think of this as Junit.tearDown()
• verify: failsafe runs another goal that interprets the results of tests here, if any tests didn’t pass failsafe will display the results and exit the build.

Configuring Failsafe in the POM:

23
24
25
26
27
28
29
30
31
32
33
34
35

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>failsafe-maven-plugin</artifactId>
        <version>2.4.3-alpha-1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

By default, the Surefire plugin executes **/Test*.java, **/*Test.java, and **/*TestCase.java test classes. The Failsafe plugin will look for **/IT*.java, **/*IT.java, and **/*ITCase.java. If you are using both the Surefire and Failsafe plugins, make sure that you use this naming convention to make it easier to identify which tests are being executed by which plugin.

In the next part of this series, I will talk about test coverage and demonstrate a method of including coverage of both Unit Tests and Integration Tests in a report that can be generated for a Maven project.