During the first run we noticed some problems.Each time Nexus started, VM would use an additional 45Mb of memory. Soon we realized that our embedder was not working right. Stopping Nexus was not removing it from memory.

Setting all fields to null after stopping helped somewhat. But now instead of 45Mb of leak we were getting 40Mb, which was still too much when we had more then 200 tests to run. It was time to profile our tests.

At Sonatype we use yourkit for profiling, so I will demonstrate how to use yourkit to profile tests executed by maven. You should be able to use the same principle with any other profile tool. 1 – First, download, get a license and install yourkit. 2 – Create a new profile to hold yourkit-related configuration:

  <profiles>
...
    <profile>
      <!-- http://www.yourkit.com/docs/80/help/agent.jsp -->
      <id>yourkit-profile</id>
 
      <properties>
        <yourkit.home>C:\Arquivos de programas\YourKit Java Profiler 8.0.13</yourkit.home>
      </properties>
 
    </profile>
  </profiles>

I put the yourkit.home in pom.xml, but you can have it in settings.xml as well.

3 – Add yourkit agent to surefire inside the profile:

...
    <profile>
...
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
              <argLine>-agentpath:"${yourkit.home}\bin\win32\yjpagent.dll"</argLine>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>

There are other ways to do this you can see here.

4 – Launch yourkit UI.

Yourkit UI

5 – Launch maven using yourkit profile: $ mvn clean test -P yourkit-profile -Dyourkit.home={your yourkit installation home}

As soon as yourkit UI detects your application, it will begin collecting profiling data.

Test running

Optional, using the API

Regular profiling is not very useful to find memory leaks on unit tests. A much more efficient approach is to take memory snapshots programmatically. To do so, we need yourkit API.

On the same profile, add this dependency:

    <profile>
...
      <dependencies>
        <dependency>
          <groupId>com.yourkit</groupId>
          <artifactId>yjp</artifactId>
          <version>8.0.13</version>
          <scope>system</scope>
          <systemPath>${yourkit.home}\lib\yjp.jar</systemPath>
        </dependency>
      </dependencies>
...
    </profile>

Note that you should not move yjp.jar to your local maven repository.

Now we can create a com.yourkit.api.Controller instance and take memory snapshos when we need to.

To avoid compilation problems during regular builds (when you are not profiling) I suggest doing that using reflection, like this:

    // profiling with yourkit, activate using -P youtkit-profile
    private static Object profiler;
 
    private static void startProfiler()
    {
        Class controllerClazz;
        try
        {
            controllerClazz = Class.forName( "com.yourkit.api.Controller" );
        }
        catch ( Exception e )
        {
            log.info( "Profiler not present" );
            return;
        }
 
        try
        {
            profiler = controllerClazz.newInstance();
            controllerClazz.getMethod( "captureMemorySnapshot" ).invoke( profiler );
        }
        catch ( Exception e )
        {
            fail( "Profiler was active, but failed due: " + e.getMessage() );
        }
    }
 
    private static void takeSnapshot()
    {
        if ( profiler != null )
        {
            try
            {
                profiler.getClass().getMethod( "forceGC" ).invoke( profiler );
                profiler.getClass().getMethod( "captureMemorySnapshot" ).invoke( profiler );
            }
            catch ( Exception e )
            {
                fail( "Profiler was active, but failed due: " + e.getMessage() );
            }
        }
    }

That way you won’t need to change the code to run in non-profiling mode.

You can see the full example in place at nexus-test-harness.

If you are still reading and are curious what was causing the memory leak on Nexus tests, we discovered that jetty creates Shutdown Hook that can hold a reference to all servers (depends on how you configure it). That shutdown hook stops jetty instances when the VM is going down. So it was just a matter of removing my jetty instances from it when I manually stopped Nexus.

The application

Let’s define a very simple application to be tested. The application will return an exit code equal to the number of command-line parameters. Running the program with no parameters will produce an exit code equal to zero, two parameters exit code two, and so on.

The application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

package org.sonatype.simpleclientapp;
 
public class Main
{
    public static void main( String[] args )
    {
        System.exit( execute( args ) );
    }
 
    public static int execute( String[] args )
    {
        if ( args == null )
        {
            return 0;
        }
 
        return args.length;
    }
}

This class is very simple to test, in the next section you will see how you can write a simple integration test using JUnit.

Creating and running integration tests

Let’s create a simple integration test using JUnit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

package org.sonatype.simpleclientapp;
 
import java.io.File;
 
import junit.framework.TestCase;
 
public class MainIT
    extends TestCase
{
 
    public void testExecute()
        throws Exception
    {
        assertEquals( 0, execute( new String[] {} ) );
        assertEquals( 1, execute( new String[] { "one" } ) );
        assertEquals( 6, execute( new String[] { "one", "two", "three", "four", "five", "six" } ) );
    }
 
    private int execute( String[] args )
        throws Exception
    {
        File jar = new File( "target/simple-client-app-1.0-SNAPSHOT.jar" );
 
        String[] execArgs = new String[args.length + 3];
        System.arraycopy( args, 0, execArgs, 3, args.length );
        execArgs[0] = "java";
        execArgs[1] = "-jar";
        execArgs[2] = jar.getCanonicalPath();
        Process p = Runtime.getRuntime().exec( execArgs );
        p.waitFor();
        return p.exitValue();
    }
 
}

Note that this test needs the packaged jar (simple-client-app-1.0-SNAPSHOT.jar), which isn’t created when surefire runs the unit tests.

To run this we need a different approach. By different approach I mean a different plugin: the Failsafe Maven Plugin. The Failsafe Plugin is a fork of the Surefire plugin designed to run integration tests.

The Failsafe plugin goals are designed to run after the package phase, on the integration-test phase.

The Maven lifecycle has four phases for running integration tests:

• pre-integration-test: on this phase we can start any required service or do any action, like starting a database, or starting a webserver, anything… you can think on this as Junit.setUp()
• integration-test: failsafe will run the test on this phase, so after all required services are started.
• post-integration-test: time to shutdown all services… you can think of this as Junit.tearDown()
• verify: failsafe runs another goal that interprets the results of tests here, if any tests didn’t pass failsafe will display the results and exit the build.

Configuring Failsafe in the POM:

23
24
25
26
27
28
29
30
31
32
33
34
35

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>failsafe-maven-plugin</artifactId>
        <version>2.4.3-alpha-1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

By default, the Surefire plugin executes **/Test*.java, **/*Test.java, and **/*TestCase.java test classes. The Failsafe plugin will look for **/IT*.java, **/*IT.java, and **/*ITCase.java. If you are using both the Surefire and Failsafe plugins, make sure that you use this naming convention to make it easier to identify which tests are being executed by which plugin.

In the next part of this series, I will talk about test coverage and demonstrate a method of including coverage of both Unit Tests and Integration Tests in a report that can be generated for a Maven project.

[media id=1 width=320 height=70] INTERVIEW TRANSCRIPT

Tim O’Brien: So, Rich could you introduce yourself?

Rich Seddon: Yeah, my name is Rich Seddon and I’ve been brought on at Sonatype for the purpose of producing a set of automated tests for the M2 Eclipse Product and the reason for doing this is that there’s a lot of performance improvements and bugs that we want to fix. But if we don’t have the automated test in place we won’t know if we’re breaking things. And it’s got to be automated because of the number of platforms that we support and the number of operating systems that we run on is very large and the manual testing process is really much, much more than a small company can possibly handle.

TO: So, can you give me a sense of what’s the difference between what you do and something like a basic JUnit test.

RS: Well, what I’m actually doing here is a full set of system tests for M2 eclipse so I actually in the test harness bring up the M2 eclipse on various platforms and drive the UI through a set of scenarios and verify along the way that no errors have occurred and that the state of the UI is as expected and the state of the project under test is as expected….you might say that what I’m doing is trying to capture, in an automated fashion, the types of usage that our users will actually go through from day to day.

TO: What kind of tools are you using for this?

RS: Right now I’m using uumm; I’m kind of using Hybrid approach. There are sort of two approaches to doing eclipse UI I testing. One of them is to drive it with a sort of UI robot where the robot pushes buttons and manipulates trees and things like that. And the other approach is sort of a white box testing approach where you bring up in a regular old unit test, eclipse, and you drive eclipse at the API level to the unit test. Both of these tests have their advantages and disadvantages.

The reason that I got attracted to this position is that, in the last few years, it seems there’s been a few sort of hybrid approaches come along one of them is called SWT POD (sp?) PPTP by eclipse and then there was Abbott. Then there’s a commercial company, Instantiations has a product called “Window Tester”. And these allow you to do both kinds of testing, UI Robot testing and API testing at the same time. What they do is they sort of record your actions as your drive eclipse through a set of scenarios and produce a JUnit test that you can then edit.

TO: Is this run as a part of the Maven build for m2eclipse?

RS: Yeah, it is run in….. integration using the Maven OSGI plugin. There’s an OSGI test runner plug-in that actually brings up the eclipse, loads in the unit test, and then within the running eclipses project runs the unit test.

TO: So, what sorts of things have you found through the testing that we wouldn’t have found if we hadn’t been doing this?

RS: Well, I’ve found quite a few things. There’s been certainly some deadlock situations that I don’t think we would have been able to reproduce. We would have got reports of them, but we never would have been able to reproduce them in-house because they don’t happen that often. But, when you have these tests running continuously and you’re capturing the output of them when things go wrong then that allows us to find intermittent bugs which is very difficult to do otherwise with manual testing.

The other thing that I’m finding though is that because I’m actually driving eclipse through an automated scenario, it forces me to pay very close attention to eclipse itself. So, I actually think even the manual testing that I’ve been doing has benefited from this process because I have to really think very carefully about how I’m driving eclipse through various scenarios in order to be able to produce these tests.

TO: I notice that sometimes the bugs that people find in a plug-in for eclipse have nothing to do with the plug-in itself. Are you testing different distributions of eclipse; like different combinations of plug-ins?

RS: I am not doing so much of that yet. I’ve just got started about a month ago, but it’s on the road map. So, I plan to have these automated tests— Right now they’re just running in Eclipse 3.4 and they’re running essentially with all the of the required plug-ins that M2Eclipse could ever never need: WTP, Subclipse and all kinds of things like that. But yes, it’s on the road map for maybe in a month or two, I’m going to start expanding the set of platforms and have different sets of plug-ins and different combinations of plug-ins. Certainly Eclipse 3.3, possibly Eclipse 3.2, IBM RAD and of course get running on different OS; These are all on the road map. And, once I’ve got the automated test running, I mean this is the advantage of the automation right, I can really easily expand it to be able to get good quality testing on all these different platforms.

TO: Well, thanks for taking the time to talk to us. And, we’ll check with you in a few months to see what progress you’ve made.

RS: All right, sounds good.

Nonetheless we’re committed to the task and we’ve hired Rich Seddon to create the testing framework and infrastructure.  Our goal is to reach the standard set by Nexus which has so many integration tests that virtually nothing regresses and we can refactor and add new features without introducing new bugs.  Unfortunately, we know from experience in our training classes and feedback from users that m2eclipse has not reached this level of quality and we are fully addressing this fact by committing ourselves to testing and quality. We probably should have pursued fewer features and more testing, but what’s done is done.  From this point forward, testing is our primary concern. We will be able to add some new features, and do some architectural work to help stabilize m2e for the 1.0 but we’re basically going to be eradicating defects and adding the testing which will facilitate the convergence of m2e with the current Maven trunk and ultimately the Maven 3.0 final release.   In this post, I’ll walk through what we’re doing at a high level and then try to layout a schedule for what’s going to happen in the next few months.

Testing Plan & Test Automation

We are working on getting a complete feature map which will allow us to accurately test the features we purport to provide in m2e. Along with the feature map, we are gathering standard use cases and creating test plans around those. For example, we are trying to walk through the chapter in Maven: The Definitive Guide on m2e and finding that we even have problems there. So, we will be mapping out the features, creating standard projects and standard workflows to test.  Throughout this process, users will have the opportunity to participate. We also want to make it as easy as possible for users to get involved and having a large number of tests can only help us in that area. We’ll be documenting the testing setup and we’ll also be making it easier for people to write tests so they can, in general, participate more effectively.

Documentation

Tim O’Brien who is responsible for the Maven: The Definitive Guide is going to pull the m2e chapter out of the Guide and create a more complete mini-book which will describe with example the features we have mapped out. Right now we have many documents in the Wiki which are hard to find and piece together, so it will be Tim’s task to organize this documentation and add to it so that we can provide a free “book” about m2e.  We’re also expanding our documentation presence to produce more cookbook-style “recipes”, and we’re rethinking the idea of the online book to provide smaller, more targetted guides for people looking to learn quickly.

Problem Reporting

One of the new features that will be released with 0.9.7 is the new problem reporting feature. Although this feature is orthogonal to the functionality directly in m2e I think this is the most critical addition we have made in some months. With one click you will be able to gather up your logs and system information and automatically submit that information to our issue tracking system. We want to accurately capture the information from problems as quickly as possible so users get correct information to us quickly. We are also making tools on our side to process these automated problem submissions so we can determine what’s wrong faster. This functionality is already present in the code base and will be in the next dev build and in 0.9.7. We took a page out of Joel Spolsky’s book:

https://fogbugz.bitvise.com/help/Articles/GetCrashReportsFromUsersA.html

Architecture

We are going to be working on the architecture of m2e to allow the clear separation of dependency management, the configuration framework, and the Maven life cycle. Right now a great number of our problems can be attributed to some muddling in this area. We made a first attempt and we see now how things internally need to change. The separation of the dependency management means that you will be able to use this feature by itself on non-Maven projects if you wish.

The configuration framework will be responsible for taking configuration in POMs and mapping that into Eclipse, but it will also be responsible for taking preferences in Eclipse and mapping those into POMs. Currently we are working on Flex integration for a client and this is the first case of having fully bidirectional configuration management. You change the POM and Eclipse preferences are changed, you change the Eclipse preferences and the POM is updated to reflect those changes.  These are what we call configurators and we will document how these work so that people can create their own bidirectional support for any tool they wish to support.

The Maven lifecycle work will focus on optimizing what Maven itself actually does. Igor is currently working on some prototypes for a limited set of use cases to make the workflow as efficient as possible. In a session with a client they reported that Maven always seems to be “pulsing”, that the life cycle is always being fired and slowing things down/consumes a lot of resources. Igor is trying to work from concrete scenarios which will help me understand what needs to be changed in Maven 3.x itself to support a Maven integration that is not so invasive and resource-hungry.

Features and Improvements

As I stated earlier, we are going to limit much of the new feature work in order to support the testing and quality aspects of m2e. But what we would like to do is capture what users are asking for in major categories like:

• Extension points for integration
• WTP functionality
• POM Editor functionality
• Tight integration with Nexus

This is the kind of information we will be looking for during the 0.9.8 iteration and, once decided, those will be the final targets for the 1.0 release. We will be looking for accurate user stories and we will be using the voting mechanism in JIRA to help us decide. We are looking to be fairly aggressive on the 1.0 schedule which we will be able to do once the testing is in place. As such we will likely time-box the releases and we’ll do as much as we can in a fixed period of time and then release. This will be easier as Rich automates more of the testing.

Convergence with Maven 3.x

As many users know we are using a version of the Maven Embedder from 4 months ago which at the time was Maven 2.1.x. In Maven, we now have a 3.x, 2.1.x, and 2.0.x branch and m2e is working from the 3.x branch. Given that we’re 4 months behind and the testing is not adequate to just slot in what we currently have on the trunk (3.x). We need to be cautious about absorbing those changes. Right now Shane and Benjamin Bentmann are almost entirely adding unit and integration tests to the Maven 3.x code base so that we can safely do this. We are about to release Maven 3.0 alpha-2 (the alpha-1 was quiet) and I imagine that around alpha-5 the testing on the Maven side will converge with the testing efforts here and we will be able to start using what’s on Maven trunk. What we are going to do is create a branch where we will integrate Maven trunk for those that want to try it but it won’t be something we suggest for daily use but will be available for the adventurous.

One of the biggest changes that we’ve made in Maven 3.x is the integration of Mercury which is the next generation of our repository management & transport APIs. We really don’t want to try and integrate Maven trunk into m2e proper until Mercury is properly integrated into Maven trunk itself. It’s a very big change which is why we need all the testing. We believe that Mercury is the only real way we can improve Maven 3.x itself so there are big changes but the long-term benefit is a big win. The artifact resolution will be deadly accurate and fast. If you want to know more about Mercury you can take a look at Oleg’s blog:

http://blogs.sonatype.com/people/author/oleg/

Project Infrastructure

We are going to be making to be making some infrastructure changes to help us organize the project better and hopefully make it easier for users to participate. Starting next week, daily builds will be available for people. As we fix issues, the builds will be available for people to try.

We are going to start using the same reporting infrastructure that we use for Maven so that we can track what users want. This is what we are currently using for Maven itself and so I would encourage users to start voting on issues and features they care about as we’ll start scheduling time to fix the issues and features with the highest vote counts. If you want to take a look at the actual reports you can take a look here:

http://www.sonatype.org/~j2ee-hudson/reports/maven-votes.html http://www.sonatype.org/~j2ee-hudson/reports/plugin-votes.html

We are going to make a big pass this week at triaging all the issue currently in our Codehaus JIRA instance because we want to reduce the issue count because we are going to migrate all our issues to the Sonatype JIRA instance. We want to do this because we have some reporting tools and plugins (like Green Hopper) which are not available at Codehaus. We also need to do this so that we can synchronize this with our problem reporting and error submission system. We are basically going to let users automatically submit issues with the data attached and we need to be able to move these issues around. Trying to move an issues from one JIRA instance to another is a real pain in the ass. We are going to take care of migrating the issues and users so there should be minimal impact to users. We will test the migration several times before we do it for real.

People also want to know about what’s happening with the move to Eclipse. We are definitely going to do this and Sonatype is a strategic development partner so this is certainly going to happen. After some discussions with folks at Eclipse we have decided to move to Eclipse after the 1.0. There are a couple reasons for this: the first is that we want to focus on testing and quality and some architectural changes will change the dependencies and this impacts the legal work we need to do. Basically we want to focus on getting a quality 1.0 out and then we’ll worry about moving to Eclipse. Moving to Eclipse incurs a lot of overhead and we just have to set some priorities and a high quality 1.0 of m2e is a higher priority for us.