This is the first production release of m2eclipse in more than a year, and it couldn’t come any sooner.  In this release, you’ll find that we’ve separated the update sites. There is now a core update site and an extras update site which contains optional components.  For more details about the installation, please read the installation instructions on the m2eclipse site.

One important note about the 0.10.0 release: you cannot upgrade from 0.9.8 or 0.9.9-dev. You must either uninstall the previous version from your Eclipse installation, or you should start with a new installation of Eclipse. The recommended version of Eclipse for this release is 3.5.1. You can download this eclipse distribution from http://www.eclipse.org/downloads.    The rest of this post details what is new and noteworthy about the m2eclipse 0.10.0 release.

What’s New and Noteworthy in this release?

• Stability – We’ve spent the last year working on stability and performance. If you are used to using m2eclipse 0.9.8, you’ll notice a remarkable performance improvement between these two versions.

• Integrated with Maven 3.0 – This release of m2eclipse includes Maven 3.0-alpha-6+. One of the primary drivers of the Maven 3.0 effort was to reimplement some of the “guts” of Maven to make it easier to embed within frameworks like the Eclipse IDE. If you are wondering what changes need to be made to your projects to use Maven 3.0, the answer is “none”. Maven 3.0 is a revolutionary upgrade that will enable the next generation of development tools, but you don’t have change a thing in your project. It should all just work.
  • Compatibility with Maven 3.0 CLI behaviour
  • Major performance improvements compared to 0.9.8
  • Full support for proxy/mirror/auth configuration as per settings.xml
  • Note for Maven 2 Users: If you need to configure m2eclipse to use a Maven 2 installation, you can do so in the m2eclipse preferences.

• Maven Project Lifecycle Mapping framework – This framework gives you the ability to customize the Maven plugins and plugin goals which are involved in your development cycle. If you need to configure the Maven Resources plugin to update resources every time your project is built within Eclipse, there is a new tab available in the m2eclipse POM Editor.
  • Developed using the new Project Configurator API
  • Eclipse project configuration and build can be fully customized for project types and individual projects
  • Implementation of plexus-build-api to allow mojos participate in eclipse incremental/full builds
  • Support for modello, plexus metadata, antlr3, build-helper, resources (from site-extras)

• Reimplemented nexus-indexer integration and repositories view – m2eclipse is very tightly integrated with the Nexus indexer, and it uses it to quickly locate dependencies and artifacts. This release adds a new Repositories View which gives you the ability to inspect, modify, and manage the Maven repositories (including your Local Maven Repository) in an easy-to-use interface.
  • m2eclipse now tracks repositories defined in settings.xml and project pom.xml files
  • New option for disabled, min, or full index details for each Repository
  • Supports the new Incremental Index Standard
  • Remote index files are cached in maven local repository and shared between workspaces, as a result workspace initialization in m2eclipse is much faster.

• Preliminary eclipse 3.6 support – While we don’t yet recommend the use of Eclipse 3.6, this release does start to add preliminary support for the platform.

When we first set out to design the external security realm (LDAP/ Crowd, etc) support in Nexus Core, we had one primary concern and that was to make it easy to integrate with systems having huge numbers of users.   Nexus was designed as a tool to be used to support the largest open source communities with thousands of developers and hundreds of projects, and like most large enterprises, these communities have settled on solutions like LDAP, Active Directory, and Crowd as a way to manage user credentials and permissions.  A secondary concern was to support any level of integration with these external security realms, specifically:

• delegating only authentication to an external server
• delegating both authentication and authorization to an external server
• delegating everything but authentication promotion permissions to an external server

These interactions were gleaned from years of experience working with customers at all levels. Some have completely centralized control over passwords and roles. Others have a situation where there’s a global repository but the roles don’t match reality, or are too hard to get updated.    We wanted to create a system that would both integrate with centralized authentication servers and allow for a sensible way to override role assignments directly in Nexus.

All of this is Core functionality works the same with all types of security realms.   We offer LDAP and Crowd support and the rest of this post will discuss the generic security model that works with every security realm that is available.  It is possible to define multiple external realms for Authentication and/or Authorization in the Server settings. This ordered list is used to validate credentials and select roles.  When a user attempts to authenticate in Nexus, it will iterate over the list until it can successfully authenticate a user.

Order Realms

For performance reasons it is almost always beneficial to move any external realms to the bottom of the list. Nexus will not check other realms once it finds a user. The first two realms are fast because the model is held in memory. The external realms tend to be slower, as they need to connect to a remote system.

Navigate to the ‘Server’ pane, scroll down to ‘Security Settings’ section, and move your external realms to the bottom of the list. (do not forget to save!)

Now that we’ve defined the ordering, lets take the full integration approach.

Authentication and Authorization

Lots of organizations have some central user management service which has the users identity, credentials, and which groups they belong to.  Nexus is able to delegate login and authentication to that server.  For authorization we map Nexus roles to your organization’s roles/groups.   If all your users are in a group named ‘users’, you can map this group to a Nexus role of the same name and assign it the desired permissions.   Once a user authenticates against an external realm, their groups will match up against Nexus roles that define the specific permissions to grant.

To map a role, navigate to the ‘Roles’ pane and select the ‘Add External Role Mapping’:

A dialog will pop up, you will need to select the security realm you are using and the group you want to be mapped.  This group is a group that is managed by the external server.   In this case, we’re mapping the “users” group in LDAP to a role in Nexus.

You then need to select which Nexus roles you want to map to your organization’s group.   Here you see the new role mapping.   To add Nexus permissions for this group, just drag Nexus roles and privileges from the Available Roles / Privileges list to the Selected Roles / Privileges list.

By mapping the external roles/groups to Nexus Roles, we are able to grant permissions to “unnamed” users. That is, ANY user that successfully authenticates against the external realm, having the group “users” will inherit the permissions granted to the Nexus “users” role. You do not need to tell Nexus about all these users ahead of time simply to assign them roles. Imagine trying to do that for 6000 users. Nexus is unique among Repository Managers with this approach.

Authentication only

Some organizations have central user management but do not maintain user groups, or the groups are unrelated to development. For this use case, Nexus can be configured to delegate the login to the server, but all the user roles are configured inside of Nexus. This provides the benefit of allowing users to use their company wide accounts while letting a Nexus administrator manage roles.

Go to the ‘Users’ pane, select a user and give them a role. In the image below, I added ‘Nexus Developer Role’ to the user ‘Maven’.

Unlike the first example, you need to manually go to each user that needs access and grant them the appropriate Nexus roles.   In this model, the Nexus administrators take on more responsibility for granting the appropriate users the appropriate level of access to the repositories they need to use.

Authentication Promotion

The last option is a combination of the two previous. The User login and list of roles are delegated to a central server, but additional roles are assigned to individual users in the Nexus UI.   In this model, Nexus administrators only need to augment roles for the few users that need to special permissions in Nexus.   For example, user Joe Coder is a developer and is part of the organization’s basic ‘development’ group. If Joe needs helps administer Nexus, all you need to do is assign him the ‘Nexus Administrator Role’.

To assign a user a specific role or privilege, click on User in the Security menu and select the user you want to add a role or privilege to.  You should see the same interface that was shown in the previous section.   While the user will have a set of basic roles mapped from the external security realm, you can add special privileges that allow a user to promote a staged release or administer the Nexus interface.

In this example it is easy to see the power of Nexus’s security framework. In a few clicks you can give your whole organization access to a Nexus instance. With a few more clicks you can promote a user to and admin, or just give them access to an additional artifacts.

Summary

The Nexus Core security support for External realms was designed specifically to manage several different integration scenarios from full external realm delegation to mixing and matching authentication and authorization so that it could work for any type of realm and organization. Wherever possible we made sure to imagine “will this work with 6000 users?”, and this is why the role to group mapping is supported and also why we don’t attempt to list all external users anywhere by default. At the lowest levels we also took care to empower realm authors to integrate in many different ways beyond just roles and permissions. A realm gets access to the exact url and http method (get/put/post/head/delete) and thus can evaluate each request using its own set of rules if desired. This is all part of our strategy to provide the most flexible and robust Repository Manager you can find.

This post focused more on the theory and mechanics of managing external realms in Nexus. If  you would like to see a more end to end solution involving securing subsets of repositories, you can see Managing OSS Forges with Nexus as most of the topics covered there directly apply to organizations of all sizes.

In addition to managing and maintaining the Maven Central repository, I also serve as the administrator for two very large forge repositories: repository.apache.org and nexus.codehaus.org. This post is going to dive into the details of the best practices that I’ve developed to maintain these very large instances. I will focus on the configuration of Nexus in this post, but if you’re interested in system level details, those are documented here.

Both of these repositories have a few things in common that have driven the design:

• there are many disparate projects deploying artifacts that require fine grained access control per project
• release repositories are synced to central
• they are the most commonly used snapshot repositories in the maven ecosystem
• the majority of users are anonymously reading the snapshots
• they are transitional repositories that replace older static repositories

They also have a few things that are very different:

• Apache is a Solaris Zone
• Codehaus is an Ubuntu Jeos VM
• Apache is using httpd for reverse-proxying and ssl
• Codehaus is using Nginx for reverse-proxying and ssl

This post contains two sections, the first covers some system-wide Nexus configuration, the second contains details about adding individual projects, along with security and staging configuration. If you are setting up a public Maven repository, this post might give you some ideas about configuration and administration issues that you’ll need to think about.

Nexus System Configuration

We want to protect all authenticated traffic, so both systems rewrite all http access to https (you can see how that’s done in the server setup linked above). However, since 99% of all traffic to these systems is anonymous, I’ve allowed the snapshot urls to poke through without being redirected.

Since I am using reverse proxies in front of Nexus, and the protocol doesn’t have a good way to tell Nexus what the inbound protocol was, I need to tell Nexus how to generate absolute urls that are used in the REST API. This is done by setting the following options in the server configuration pane:

The systems are configured with just two hosted repositories: releases and snapshots. Both systems are transitional, meaning that projects elect to convert at a convenient time. To support this, I proxy the old snapshot repository and aggregate it with the locally hosted snapshot repo. When you hit http://repository.apache.org/snapshots or http://nexus.codehaus.org/snapshots, you’re hitting this group and it appears as one repository. We also have a staging group that is used to aggregate all staging repos that haven’t yet been promoted. Here’s what the repository list looks like:

One benefit to using Nexus in these forge setups, is that we are able to configure rules that automatically check staged artifacts before they can be promoted. This includes things like validating the pgp signature is present and signed with a publicly accessible key, looking for sources and javadocs, validating the pom, etc. This is one way we are helping to improve the data in Central, by helping to correct it right at the source. Since these rules are tied in to the Staging support, we want to disable the ability to deploy directly to the releases repository. This is done by setting the repository url to be read-only:

I also have configured the following jobs:

• Configuration Backup – Backs up the Nexus Configuration files. I have it set to run daily and to keep 10 days of backups
• Publish Indexes – This packages the internal real-time indexes into a format that is consumable by downstream Nexus’ and M2eclipse users (other tools consume this data as well). I have this set to run daily.
• Purge Proxy Artifacts – Since we’re transitional and proxying the old, static snapshot repositories, I have configured a task to evict items that haven’t been requested in more than 10 days. This just reduces the disk consumption on the repo. If a file is re-requested later, it will be retrieved again from the proxy on demand.
• Snapshot Cleanup – We want to enforce best practices and keep snapshots moving forward. The cleanup task is set to keep a maximum of 3 timestamped snapshots for each artifact for a minimum of 10 days. All snapshots for an artifact are also purged when a release is promoted.
• Empty the trash – All delete operations in Nexus never actually delete, they just move files to a trash folder. This is a security net in case you misconfigure a cleanup task, or simply make a mistake in the ui (like dropping a repo you meant to promote). We keep on top of the trash by scheduling it to run once a week. New in 1.4 is the ability to purge things from the trash only after they have been deleted for x days. I’ve set this to hold things in the trash at least 7 days. This gives projects more than enough time to detect any issues and recover the artifacts.

Project Specific Configuration

Each project needs to have access to only their own artifacts. Nexus supports two different ways to handle the security separation. If you want to read more about the two modes read my previous post:“Optimal Nexus Repository Configuration” .  We have chosen to manage the forge by partitioning a single pair of repositories. The next few steps show how this is done.

First, we define a new Repository Target for this project’s artifacts. Don’t worry if you’re not a regexp genius, wildcards are very easy, and we let you define multiple regexps so you don’t have to figure out more complicated and/or expressions. In the image below, I’ve created a new target called “org.codehaus.org” that will contain all artifacts in the paths /org/codehaus/mojo and below.

Now that we’ve defined our “bucket” of artifacts, we need to create some permissions that are associated with it. You’re probably thinking “create permissions?” Yes, see the Repository Target is a generic concept that lets you arbitrarily group artifacts, but notice we haven’t yet associated the target with any repositories.

NOTE: The logic behind this approach is that you may want to grant people read access to all org.foo artifacts, but what if you only want them to see artifacts that have been promoted and not things that are still being staged?

In this case, we want to grant CRUD (Create / Read / Update / Deleate) to SNAPSHOT artifacts, but only CRU to releases. (Admins only are allowed to delete releases, this prevents problems once things are synced to Central). I will do this in two steps as shown below. First create a set of permissions that link the org.codehaus.mojo Repository Target to “All Repositories”:

Then create a set or permissions that only apply to the hosted Snapshot repository:

Both the Apache and Codehaus forges use the Staging support in conjunction with Staging rules to validate the integrity of releases.

NOTE: Nexus staging is unique in that it’s entirely controlled from the server side, which means Admins can adjust as needed without changing the poms. It’s also designed so that all projects use a single url for deployment that is abstracted from the repository, which provides two benefits: 1) you can change the repository hosting artifacts without changing poms and 2) you can specify the distributionManagement url in just one place, reducing the errors. For example, at Apache we have a parent pom that contains all the logic a project needs to be staged.

We control this in Nexus by creating a Staging Profile. Fortunately the profile reuses the Repository Target we defined earlier:

Not shown are the settings that let you set the validation rules and who should be notified at each promotion step.  Now that we have defined the staging profile, the system automatically created a few new permissions that let you specify who is allowed to stage, drop and promote these artifacts. We want to grant this permissions to users and this is done via roles.

The Codehaus repository is linked to their LDAP system. Nexus takes a unique approach that lets you easily grant access to dozens of users without having to configure each user in the system. We do this by allowing you to “map an external role” and then grant Nexus specific permissions to any user that has this role.  To do this, navigate to the Role pane and select the Add External Role Mapping as shown below:

This brings up a dialog where you can select the external realm (you could have multiple realms), and then you will see a list of all the roles known to the external system. Here I’m selecting “mojo-developers”.

This creates a new role in Nexus with an id matching the external system id “mojo-developers”. Any user authenticated that has this role in the external user account will automatically be granted these permissions.

I now select the roles and permissions I want to grant. Specifically, I grant the following:

• Staging: Deployer (org.codehaus.mojo) - This is a role that was created when I setup the profile. It contains the basic set of permissions needed to allow a user to view, stage, close and drop org.codehaus.mojo staging repositories (only)
• UI: Staging Repositories – This lets the user actually see the staging view, without this they couldn’t see what they staged.
• org.codehaus.mojo – All – Here I’m granting Create, Read and Update for all matching artifacts (remember these are the permissions we created above that apply to all repositories)
• org.codehaus.mojo – snapshots: Here I’m granting delete, but only for the org/codehaus/mojo artifacts in the Snapshot repository.
• Staging: Profile org.codehaus.mojo – (promote): The Staging: Deployer xxx roles give the ability to stage, but not the ability to promote. This permission may be granted to managers, qa, or PMC etc as appropriate. Here we’re letting developers stage and promote their own artifacts.

And we’re done. Notice that I didn’t need to go grant permissions to every user, and I didn’t have to put the users into groups. This is the power that Nexus provides in user management. This is core functionality that applies to any realm you may have, not just Nexus Professional ones.

Now, to illustrate some more power of this approach, what if org.codehaus.mojo later starts managing artifacts under org.mojo? I don’t have to redo everything here, I just extend the Repository Target “bucket” by adding “./org/mojo/.” and instantly all the permissions, staging profiles, etc apply to the new groupId. This has definitely saved me many times at Apache with the Webservices Projects… they have more groupIds than I can count, but they all map back to the same external role. Each time a new one comes along, I just add it to the target and I’m done.

Automating Nexus Administration via REST

This is the process we’ve ironed out over several months. I definitely don’t go through this UI clicking every time. Since Nexus has a full REST API (the UI is just a REST client built in JavaScript), we have developed a set of command line tools that take a few simple inputs like groupId, external role id and project name and automates all of this configuration via REST calls automatically. You can see those tools here and they make a great example of how to integrate Nexus into the DNA of your organization.

In “A Tale of Two Repository Managers” John Smart compares Nexus to Artifactory, and covers some of the more well known features like Staging and Security.  I wanted to emphasize a few more of the other features that are often overlooked.

Most of our users download the tool, install it, and use the most straightforward features: simple proxy repositories, hosted repositories, and repository groups.  We’ve gone out of the way to make Nexus intuitive, but I often wonder if enough people know about some of the features offered as part of the base project.  Here are some of the features I would have highlighted in any comparison.

Mirrors / Repository Metadata Standards

We manage the Central Maven repository: a free public resource used by millions of developers every single day.   This is a big part of what we do on a daily basis, and so we are acutely aware of the bandwidth being consumed serving the canonical, central repository to all the Maven, Ant, and Ivy users out there. Nexus is the only repository manager that is able to deal with mirrors in a true “mirror” fashion. Using the repository-metadata.xml if it exists, or by manual entry otherwise, Nexus is able to discover known mirrors of a repository. So when configured, Nexus is able to use a mirror that may be geographically closer to retrieve artifacts, while still falling back to the canonical repo if artifacts are missing and/or to validate checksums and pgp signatures.  (http://www.sonatype.com/people/2009/03/new-feature-in-nexus-13-mirror-support/) This can greatly improve proxy performance without any of the downsides associated with using a mirror that could be slightly stale.   No other repository manager currently does this.

OpenSearch

In the searching arena, Nexus also supports the OpenSearch standard, allowing searching to occur directly from the search box in your browser (http://www.sonatype.com/people/2009/06/demonstration-of-nexus-opensearch-integration/)    If you use Firefox, how often do you load up “http://www.google.com” to perform a search?   It is a good chance that you use the search box that is integrated into your browser.    This same search box can also be configured to search a Nexus instance.    No other repository manager offers Open Search integration.

Indexer Standard

The Nexus Indexer format is the standard that all repository managers and IDE integrations use to fetch information about the contents of a remote repository. So while it’s true that the search in Nexus isn’t as fully featured as we would like, we have spent the time instead focusing on interoperability at the index level. The standard has been updated to break the dependency on Lucene 2.3, and to support incremental index downloads. As it stands today, Nexus is the only repository manager that can produce (not just proxy) indexes that are compatible with all the Maven IDE plugins and other repository managers. We find that most users would rather do the searching within their IDE and so we are empowering those users directly via the index integration. That’s not to say we don’t have some revolutionary things up our sleeve in the Nexus Search, but I’ll save that for another forum

All repository managers use the repository index standard that is set by the Nexus Indexer. All repository managers benefit from the time Sonatype has invested both in the Maven project itself, the development of an open source Nexus Indexer, and our continued efforts to reduce the bandwidth burden on Central as it serves an updated repository index to a world of developers using tools that interoperate with the formats we helped design.

Layout Mediation

Nexus is designed to be repository layout agnostic, while “Artifactory is a Maven 2 only repository manager by design.” Although we got some things wrong internally in the initial Nexus releases, things have been markedly improved. Nexus is able to support Maven1, Maven2, Eclipse P2, Eclipse Update Site, OBR, and most recently RubyGems. In many cases, these formats can be converted on the fly from one to the other. For example, you can expose an OSGI Bundle Repository containing every bundle in a Maven 2 repo (yes, even Central) in just a few clicks. If you work in an environment with multiple repository standards, Nexus is your choice.   No other repository manager even comes close, our core engine with layout agnostic and we’re building a tool that will serve as a virtual “Tower of Babble”, converting between repository formats once thought incompatible.

Eclipse Distribution Management

One of the most overlooked features of Nexus Pro is the ability to proxy and aggregate Eclipse P2 and Eclipse Update sites. That means that all the benefits you gain by proxying and aggregating Maven repos are directly applicable to your Eclipse IDE users: Bandwidth savings, Time savings, single url management, etc. If you would like to understand more of those benefits, John Smart did a great job of enumerating them here: http://today.java.net/article/2010/01/04/maven-repository-managers-enterprise

Extensibility: Plugin API and REST Services

Nexus has a plugin model and associated maven-archetype to make it easy to extend. Users are already adding new features like security realms, repository types and new UI capabilities.    If you take time to understand the architecture of Nexus, you’ll see that much of the functionality is implemented as a plugin.   Plugins can affect the request lifecycle in Nexus, they can define new repository types, and add new capabilities to the underlying platform.   Plugins can also customize the user interface and add new REST services.

Everything the Nexus web interface does is the result of some interaction with a REST service which you can access with your own custom Nexus plugins or user interface.   If you have an idea for a novel approach to using Nexus or if you want to build your own custom administrative scripts that automate certain tasks, you can do so without having to deal with user interface issues.   Nexus is the only repository manager that is built on a foundation of open REST services.   Our REST services are not just an afterthought, a feature to list in a feature matrix.   The REST services *are* Nexus, and you can take these core REST services and create novel tools and interfaces.

We were joking the other day that we’ve made Nexus so extensible, you could easily modify it to support the distribution of digital media.   You could even use the metadata search feature to capture, index, and track your iTunes library with a custom repository type.    While the discussion was facetious, the conclusion is accurate.   We’ve been focused on using Nexus to develop solid, enterprise-class tools to manage binary artifacts and the metadata they are associated with.   You could use Nexus to keep track of just about anything.

Summary

In short, Nexus is more than just a Maven repository manager. It’s part of a much larger ecosystem of tools that work together with defined standards that go from the command line build, to the CI system, the IDE and to production roll out. Nexus is developed and maintained by many of the leaders in the Maven arena, is backed by full time QA and support staff, and has a full length free book available for printing at cost.

The team that develops Nexus is also concerned with much more than a repository manager.  The decisions we make about product features and direction are influenced by progress on the various open source projects that we participate in.   Sonatype does more than just talk about repository managers.  Through our work in the open source community, we are participating in open communities, and giving back as much as we can to the communities that support us.    In addition to working on Nexus 1.4 over the past year, our engineers have served as release managers for new versions of Maven, and they have been working round-the-clock to get projects like Maven 3, the newer polyglot extensions to Maven, and Maven shell out and available for everyone to use.

We’re actively involved in open source, and I’d like to think that this counts for something when you are selecting a repository manager.   Choose Nexus Open Source or Nexus Professional.

In parts one and two of this series, I addressed several concerns raised on the Jboss wiki. In this post, I address several more topics. No separation of build instructions vs. general project information

both the set of build instructions (lifecycle) and general information about the project and dependency information. IMO this is a flawed model. Instead of having a single file, there should be some separation between the build instructions and the project information needed in the repository….When a POM is used as a dependency (not a parent) from the repository only certain information is useful to the calling project. Information like the project identifiers (groupId, artifactId), general project information (name, description, scm, etc). Other information is ignored: everything in the build and reporting sections, the distributionManagement…

I’ve had the same thoughts recently as we begin to think about dealing with POM model changes. Many of the changes we would want to make specifically only apply to the <build> section. Having separate files for this information is one way to solve the model / parsing problem, but it doesn’t solve it completely, since we also want to extend things in the dependency section like scope, global excludes, etc.

In hindsight, the separation may have made our work today easier, but separating it now doesn’t push us forward in a meaningful way by itself, and now we have history and convention to consider.

If a POM is used in a parent/child relationship, then the this information will be used. The problem with the parent/child model that Maven uses, is that build information can only be picked up from a single source. This means that I can’t get some build configuration from one POM and other information from another.

Yep, I addressed this in part one, but in Maven 3.x we will support the concept of “mixins” which is a way to introduce composition to the pom. This will completely solve this concern.

The project configuration (POM) used locally is not necessarily the same as what should be released/deployed. For example, in a local build configuration we might want to avoid using a specific version of a dependency and instead just let the dependency resolver get the latest released version. In the repository however, the released and/or deployed POM should always specify the version of the dependency that was used during the build. Maven recently has some workarounds for this problem such as the versions-maven-plugin which helps to keep dependency versions up to date, but the problem still exists that I have to manually use this plugin to update the dependencies. In an ideal build tool, this functionality would be automatic.

I tend to agree here as well. I’ve spent quite a bit of time considering how to deal with this recently, but every time I think it’s figured out, I find a reason it won’t work.

The implication here is that in your build you define a version of some dependency, let’s choose a range like anything between 1.0 and 2.0 not including 2.0 ( that would be [1.0,2.0) in the pom) . At the time I build my module, I resolve against 1.7 for some unimportant reason. My pom is deployed to the repository with a dependency version of “1.7″. Life seems fine.

Someone else comes along and depends upon my JAR. He wants to use 2.1, since the deployed POM now says 1.7, my range information has been lost and Maven doesn’t know that the developer has really said that it only works with 1.x.

In summary, replacing the requested version with the version resolved at build time results in a loss of fidelity that may affect consumers of my JAR.

One solution to this may be to include the resolved version alongside the requested version in the deployed POM, but we’d have to think through what the resolution strategy would do differently in this case.

No easy way to add plugin configurations to a lifecycle

An… example is the compile vs. test-compile goals, these two goals are the same logic with a slightly different configuration. Ant has a single compile task and compile vs. test-compile is just a different configuration of the same task.

It is technically possible for us to change the lifecycle configuration to allow the plugin config to be defined, and you’re right this would allow some plugin goals to be collapsed down into one.

However, we would need to think through the implications of tying the config into the lifecycle when it comes to plugin versions. If the lifecycle defines some default configuration that changes between plugin versions, what happens when the user chooses that version in their pom?

Currently you almost never have to change a lifecycle, and you especially don’t have to do it to support different plugin versions. I feel like linking the lifecycle and plugin this tightly would cause more problems than it solves.

After all, if you code your plugin correctly, making another goal with slightly different default parameters is a trivial endeavor.

Limited Access to the Build Lifecycle.

There is no simple way in Maven to add a new phase to the build lifecycle.

Absolutely and I think that’s a benefit of Maven not a problem. If you have any Maven build, you can sit down and know immediately how to run tests, compile, package etc. You don’t have to go read the build file to see if it’s “clean-test, test-clean, jar, package” etc.

Every Maven build has the exact same set of targets and it should always stay that way.

No way to handle renamed artifacts and/or link two dependency resolution IDs

There are many situations where it would be useful to link two dependency IDs to tell Maven that the two IDs refer to the same artifact. For example when a dependency is moved to a new groupId, the old groupId/artifactId is treated as a completely separate artifact. The old groupId/artifactId must then be excluded from each location where it appears in the dependency tree. Maven should have a way to link these together and ignore any occurrences of the old groupId/artifactId.

This has always been supported with the relocation element. See:

• http://maven.apache.org/guides/mini/guide-relocation.html
• http://maven.apache.org/ref/2.2.1/maven-model/maven.html#class_relocation

Maven only allows a single source directory per project

In some cases it’s useful to separate source code into multiple directories that have different compilation parameters. Unfortunately this is not allowed in Maven.

While it’s true that the POM model only allows a single source directory in the POM, Maven internally supports as many source folders as you want.

Typically additional source folders are needed in cases where code is generated and you want to keep that separate from your code that needs to be in SCM. The plugin that does the generation is responsible for adding the additional folder directly into the model in memory.

The build-helper plugin has goals to allow you to attach arbitrary folders if you need them. Perhaps when we solve the POM model problem, we can consider allowing multiple source folders directly in the POM. It’s probably already possible to do this with the polyglot stuff Jason has been working on.

Stay tuned for part 4 to see the conclusion.

In part one of this series, I addressed several concerns raised on the a Jboss wiki. In this post, I specifically focus on the issues raised with the release plugin.

The wiki states:

The release process is not reliable Maven projects use the maven release plugin to help automate the release. Unfortunately we have experienced numerous problems with this plugin. The problem is not so much the bugs in the plugin….I think a lot of the problems with the release plugin stem from two things…

Before we dive in further, the idea behind the release plugin is that most releases follow these general steps:

1. Build and verify that the tests pass
2. Tag it
3. Checkout and build the tag to be sure what you tagged is in fact correct
4. Publish the artifacts of the build for testing

The release plugin handles steps 1 and 2 in the release:prepare goal. Steps 3 and 4 are done by release:perform.

Maven’s normal development process adds a few extra steps in there regarding the conversion of the versions from SNAPSHOTS to releases, but the above steps are generally what is being automated.

In addition to automating the testing and tagging process, the plugin also makes sure you don’t:

• have uncommitted changes
• depend on any snapshots

It’s worth noting here that there is no requirement to use the release plugin. It’s meant to encapsulate best practices and generally make life easier for maven projects, but if it doesn’t, don’t use it. It’s a plugin and the logic isn’t baked into Maven core.

One is the svn tag and checkout. Because these are not done prior to the start of the release process, if there are any problems with these steps they will block a release. They can be difficult to debug because the release plugin forces the whole process to restart in order to solve the issue. Manually doing a tag and checkout is much easier because issues with svn can be resolved by themselves outside of the release plugin automation process.

If you were to manually tag and checkout something, I presume you would have built it first to make sure your tests pass? You’re asking the release plugin to pickup at step 3 above and let you do steps 1 and 2 manually, at that point you might as well not even use the release plugin and just run ‘mvn deploy’.

The release plugin has a state machine that keeps track of the last step that successfully passed. This means it doesn’t require you to start back at the beginning if something fails during the prepare process. It is true that if errors are encountered during perform that require changing code, you will have to redo the tags. There is a rollback goal to handle this.

The second major issue comes from changing the version numbers in the pom at release time. The POM versions are changed automatically by the release plugin. This seems like a good thing at first, but it can cause unforseen problems in multimodule projects that need to resolve inter-module dependencies. Sometimes dependency resolution errors are found only at release time because the new versions do not yet exist in the maven repository. This is another area where a more manual process would be preferred in many cases because the multiple step nature of the release plugin can make debugging difficult and time consuming.

The changing of the version numbers isn’t directly the problem here but it is a trigger. There are some weird interactions with some plugins and the reactor that can cause them to not resolve things that aren’t in the local repository. This has been significantly improved in Maven 3.x but it’s impossible to say if all plugins will work correctly yet. Some, like the dependency plugin may need updates to find things in the reactor.

Generally speaking though, it is possible to make your build behave correctly in a perfect clean-room scenario (i.e. where no snapshots of this project exist yet), and if it does that, releasing won’t be a problem. If it doesn’t, that’s an indicator that the build might need a few tweaks.

For all maven projects I currently work on at Sonatype and at Apache, the release plugin works perfectly. The key to this is that most of these projects were setup originally as Maven2 projects and thus follow all the best practices.

However

That said, I’m also not a huge fan of the current release plugin and have had troubles with it on other projects, so I totally sympathize. My main beef is that while the steps in the release plugin are appropriate for most projects, sometimes they just don’t fit yours.

I would like to see the release process become more like a lifecycle, and each individual action bound like a plugin to a phase. Then you can totally rewire the steps in the order that makes most sense for your project.

I have built the equivalent of the release steps as enforcer goals over the years working in the direction of a pluggable release lifecycle. It is possible today with the enforcer to string together all the validations into your own release process, completely bypassing the release plugin all together. That leaves you to manually tag, checkout and manipulate the versions, but sometimes that’s just what you need/want to do.

All that’s needed is someone to put the time in to create this new release plugin.

Part 3 will address the rest of the issues raised.

I was doing a little digging for some old emails recently and realized that it’s been over a year that oss.sonatype.org has been available for free Maven repository hosting of Open Source projects.

The first was plexus, naturally since Maven and Nexus are both currently built on top of plexus.

Our first real external “customer” on oss.sonatype.org was the Jetty project, followed by Appfuse and OpenSymphony.

We now have over 81 projects actively using the system to host, stage and sync their artifacts to Central. The repository is hosted on a machine at Contegix that is racked up next to the Maven Central repository, making it possible to sync new releases over hourly, not to mention that it is also extremely reliable.

If you have an Open Source project looking for a better way to stage your artifacts and promote them to Central go here to get started. We’ll even help you migrate your existing artifacts.

We also provide free Nexus Professional licenses to projects or forges that would rather run their own system. You may request a license by filling out this form: http://sonatype.com/products/nexus/professional/oss

I was recently pointed to a list of Maven Problems maintained on the JBoss wiki. This is actually a great summary of areas to improve, and I’m not surprised that the list is very close to the list of issues that the Maven community has been working to address over the past few months. In this post, I’ll take this chance to address each of the points raised on the JBoss Wiki.

Issue: No easy way to test multiple dependency configurations

Some projects, such as Hibernate would benefit from an easy way to configure multiple dependency configurations.  This is especially useful for setting up multiple test configurations.

Being able to depend upon, or import, a specific set of dependencies was introduced in Maven 2.0.9 with a new “import scope” that allows you to import the dependencyManagement section from another pom. This works in some cases, but it was essentially a hack to introduce new functionality without changing the POM format.

A little background info: Maven 2.x has always used a POM model version 4.0.0, and the parsers in 2.x are not capable of understanding newer versions. This is a pretty significant issue that the team is well aware of since moving to a new model is required for most of the powerful things we need to do going forward. For now, it’s not clear how to do this in a way that allows projects built with a new model to be consumed by users of older versions of Maven. We have many ideas and could fill a whole blog post on that topic alone. Lets just say this is an unresolved blocker for solving issues like this.

What we specifically need to do to address this problem is to create a new first class element called TargetPlatform (to steal a term from the Eclipse world). This would allow better support of groups of dependencies that represent the runtime environment. Without changing the model, it isn’t possible to define these platforms in a useful way.

Issue: Maven builds are significantly slower than Ant builds.

A full build of the App server would take about 5-6 minutes and an incremental build about 1-2 minutes.  With Maven, the build takes about 10-12 minutes for a full build and 3-5 minutes for an incremental build.  Having fast incremental builds is very important to facilitate development.

I don’t think this is an apples-to-apples comparison. Maven by default executes your unit tests, which naturally takes longer. It’s been a while since I significantly used Ant, but the last I knew, it was not exactly incremental either.

Maven 3.x takes a significant step forward in performance. We now have a specific suite of performance tests that enable us to measure the impact of changes on the build performance. Because of this, Maven 3.x is anywhere from 25% – 2x faster than 2.x. Most of these changes have come about as part of improving the performance of the embedder and plugins for the upcoming M2eclipse .9.9 release. Specifically, an incremental build interface has been defined and several key plugins are now updated to use it. This means that in M2e builds, we can tightly integrate with the Eclipse builder and provide a list of things (files) a given plugin depends upon, and then communicate back the files that plugin has actually touched. This is the first step in being able to support full incremental build from the CLI, as many more plugins will need to implement the new interface before builds can become truly incremental like a good make based build can be.

Issue: Building a subset of modules

I definitely missed this from my Make days and long ago wrote a proposal for it. Dan Fabulich picked this up and implemented it first as a plugin for 2.0.x and then in the 2.1.x core. You can read about these Advanced Reactor Options in Maven: The Complete Reference.

Issue: Maven is limited for of sharing configuration

For example, in a multi-module build I might want to have a set of properties that are generated before any module build.  This could be something like generate a timestamp, get the current svn revision, etc.  In Maven these properties must be set separately for each module.  If there was better support for shared module tasks, the properties configuration could be run before any module and the properties would easily be available to all the modules.

Currently the only way to share configuration is via inheritance. For example in Apache projects, we have a pretty robust toolchain for building our releases. If you want to use this in your project, you would have to copy all the associated XML since it wouldn’t make sense to inherit from our POMs.

We are working on a concept called “mixins” which is essentially composition in the poms. This will allow you to deploy and then later depend upon snippets of POM XML to compose your build.

Issue: One artifact per project rule

Maven has a general rule that there should be only a single artifact per project.  This is important to maven dependency resolution, because there is only one set of dependencies in the POM.  If multiple artifacts are generated in by a single POM (this commonly done by the assembly plugin) then there is no way to list a separate set of dependencies for the non-primary artifact.  Maven projects also often include a -sources jar and a -javadocs jar.  There is no intuitive way to generate a source jar for a secondary project jar.  For example if my project includes a “-client” jar, then I might also want something like a “-client-sources” jar.

The idea here is to encourage separation of concerns. Traditionally people lump API, client and, server code into a giant build that spits out multiple artifacts. The way this is intended to work in Maven  is that each one is it’s own project. For the most part, this holds up under all kinds of scenarios.

However, I have seen some issues when tools like Axis are used to generate the client and server directly from some other definition. There isn’t currently a clean solution for this when the resulting JARs have different dependencies. We’ve kicked around a few ideas, but no solid proposal exists yet.

I’ll address the rest in part two of this post.

The Nexus Professional 1.4 release offers a wide array of features

• Proxy Repository Browsing – With Nexus 1.4, the local cache and logical index views have been separated into separate tabs. We found that the previous single tab with a combo box to select the source was confusing.
• Publishing Web Sites to Nexus Professional – Nexus Professional 1.4 provides you with a WebDAV endpoint for publishing a web site. You can configure a Site repository that can be used as a publishing destination for project documentation. This means you don’t have to worry about providing some alternate solution to host your reports. The full support of Nexus security applies to this new type of repository, so you can control access at a very fine grained level.
• Repository Configuration Changes
  • Fine-grained control of Redeployment for Hosted Repositories: Nexus 1.4 provides administrators a simplied way to control how a hosted repository deals with the redeployment of artifacts. You can configure a repository to allow for the redeployment of previously deployed artifacts, allow for one-time deployment, or to provide a read-only interface for clients.
• Improvements to the Staging Plugin – The staging plugin had numerous improvements in the 1.4 release to increase usability and provide new functions for staging artifact bundle and verifying that staging repositories follow user configurable rule sets.
  • Support for Staging Rulesets: Nexus Professional 1.4 provides administrators with the ability to define a set of rules to apply to staging repositories before they can be promoted. The 1.4 release can validate that staging repositories contain valid POMs, valid PGP signatures, javadocs, and sources for all artifacts. The rules are pluggable and we expect to add more rules in the near future to support the Apache repository and our OSS hosting repository
  • Support for Uploading Artifact Bundles: The staging plugin now accepts artifact bundle uploads. Artifact bundles are archives which contain one or more associated artifacts, they are used to publish artifacts to the Central Maven repository, and you can use Artifact bundles to validate artifacts uploaded to Nexus.
  • General Usability Improvements in the Staging Plugin: This release of the Staging plugin focused on usability, the Staging plugin is full of improvements that make the user interface more intuitive and easier to use.
• User Account Plugin – The User Account Plugin in Nexus Professional gives unauthenticated Nexus users the ability to sign-up for a Nexus account. When this feature is enabled, a new user would click a sign-up link, fill out a simple profile form, read a captcha, and then activate a new account via an email confirmation message. Nexus Administrators can configure the default roles and permissions that are granted to newly signed up users.
• Repository Summary Panel – The repository summary panel provides statistics and configuration information for a specific repository. Users can consult the repository summary panel to gather the necessary distribution management settings for Maven configuration.
• Security Improvements – Many improvements to the user security model. In general, it is now easier to configure custom role mappings for externally managed users, and Sonatype has paid close attention to the user interface for managing users and roles. It is easier than ever to configure and secure a Nexus repository.
  • New User Role Tree: Click on a user and then click on the user role tree to see how each role contributes to the permissions for a particular user.
  • New User Privilege Trace feature: this features allows Nexus administrators to pinpoint which roles contribute which permissions to a particular user. While the user role tree provides an intuitive interface that lists role in a hierarchy, the privilege trace panel under user administration provides an alternate view. Click on a particular permission to find the roles contribute that permission to a user.
  • New Role Tree: Since a Nexus role can consist of both roles and privileges, we’ve provided an intuitive tree browser that allows an administrator to browse the hierarchy of roles and privileges associated with a Nexus Role.
  • Fine-grained control of View Repository Privilege: Nexus added the ability to configure a role to prevent users from browsing particular repositories. This is used to provide a cleaner view to users, for example to show them only groups they use via Maven and not confuse them with all the repositories aggregated by that group.
• Integration with Atlassian Crowd – Atlassian Crowd is a capable user and directory management system that can consolidate authorization and authentication to a central server. Nexus Professional’s Atlassian Crowd plugin provides seamless integration between Nexus and Atlassian’s Crowd server.
• Automated Nexus Error Reporting – Nexus 1.4 ships with an automated error repository system which can be configured to report Nexus exceptions and errors to the Nexus Issue Tracker. If configured, the system will send data to Sonatype’s Jira instance. The information contained includes the configuration (all passwords are obfuscated) as well as a file list of the repositories and exception traces. All of this data is encrypted using public-key cryptography so only Sonatype can view the contents. We expect that this information will allow us to further refine the stability of Nexus.
• Upgrades to the Nexus Book
  • A new chapter on Nexus Best Practices.
  • A new chapter on publishing web sites to Nexus.
  • Over 100 corrections and clarifications.
  • Over 80 new figures and diagrams.
  • Addition of a New Nexus Book Cover with the Nexus Logo
  

On larger projects, additional dependencies often tend to creep into a POM as the number of dependencies grow. As dependencies change, you are often left with dependencies that are not being used, and just as often, you may forget to declare explicit dependencies for libraries you require. Because Maven 2.x includes transitive dependencies in the compile scope, your project may compile properly but fail to run in production. Consider a case where a project uses classes from a widely used project such as Jakarta Commons BeanUtils. Instead of declaring an explicit dependency on BeanUtils, your project simply relies on a project like Hibernate that references BeanUtils as a transitive dependency. Your project may compile successfully and run just fine, but if you upgrade to a new version of Hibernate that doesn’t depend on BeanUtils, you’ll start to get compile and runtime errors, and it won’t be immediately obvious why your project stopped compiling. Also, because you haven’t explicitly listed a dependency version, Maven cannot resolve any version conflicts that may arise.

A good rule of thumb in Maven is to always declare explicit dependencies for classes referenced in your code. If you are going to be importing Commons BeanUtils classes, you should also be declaring a direct dependency on Commons BeanUtils. Fortunately, via bytecode analysis, the Maven Dependency plugin is able to assist you in uncovering direct references to dependencies. Using the updated POMs we previously optimized, let’s look to see if any errors pop up:

$ mvn dependency:analyze
[INFO] Scanning for projects...
[INFO] Reactor build order:
[INFO]   Chapter 8 Simple Parent Project
[INFO]   Chapter 8 Simple Object Model
[INFO]   Chapter 8 Simple Weather API
[INFO]   Chapter 8 Simple Persistence API
[INFO]   Chapter 8 Simple Command Line Tool
[INFO]   Chapter 8 Simple Web Application
[INFO]   Chapter 8 Parent Project
[INFO] Searching repository for plugin with prefix: 'dependency'.

...

[INFO] ------------------------------------------------------------------------
[INFO] Building Chapter 8 Simple Object Model
[INFO]    task-segment: [dependency:analyze]
[INFO] ------------------------------------------------------------------------
[INFO] Preparing dependency:analyze
[INFO] [resources:resources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:compile]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:testCompile]
[INFO] Nothing to compile - all classes are up to date
[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING]    javax.persistence:persistence-api:jar:1.0:compile
[WARNING] Unused declared dependencies found:
[WARNING]    org.hibernate:hibernate-annotations:jar:3.3.0.ga:compile
[WARNING]    org.hibernate:hibernate:jar:3.2.5.ga:compile
[WARNING]    junit:junit:jar:3.8.1:test

...

[INFO] ------------------------------------------------------------------------
[INFO] Building Chapter 8 Simple Web Application
[INFO]    task-segment: [dependency:analyze]
[INFO] ------------------------------------------------------------------------
[INFO] Preparing dependency:analyze
[INFO] [resources:resources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:compile]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources]
[INFO] Using default encoding to copy filtered resources.
[INFO] [compiler:testCompile]
[INFO] No sources to compile
[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING]    org.sonatype.mavenbook.optimize:simple-model:jar:1.0:compile
[WARNING] Unused declared dependencies found:
[WARNING]    org.apache.velocity:velocity:jar:1.5:compile
[WARNING]    javax.servlet:jstl:jar:1.1.2:compile
[WARNING]    taglibs:standard:jar:1.1.2:compile
[WARNING]    junit:junit:jar:3.8.1:test

In the truncated output just shown, you can see the output of the dependency:analyze goal. This goal analyzes the project to see whether there are any indirect dependencies, or dependencies that are being referenced but are not directly declared. In the simple-model project, the Dependency plugin indicates a “used undeclared dependency” on javax.persistence:persistence-api. To investigate further, go to the simple-model directory and run the dependency:tree goal, which will list all of the project’s direct and transitive dependencies:

$ mvn dependency:tree
[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'dependency'.
[INFO] ------------------------------------------------------------------------
[INFO] Building Chapter 8 Simple Object Model
[INFO]    task-segment: [dependency:tree]
[INFO] ------------------------------------------------------------------------
[INFO] [dependency:tree]
[INFO] org.sonatype.mavenbook.optimize:simple-model:jar:1.0
[INFO] +- org.hibernate:hibernate-annotations:jar:3.3.0.ga:compile
[INFO] |  \- javax.persistence:persistence-api:jar:1.0:compile
[INFO] +- org.hibernate:hibernate:jar:3.2.5.ga:compile
[INFO] |  +- net.sf.ehcache:ehcache:jar:1.2.3:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.0.4:compile
[INFO] |  +- asm:asm-attrs:jar:1.5.3:compile
[INFO] |  +- dom4j:dom4j:jar:1.6.1:compile
[INFO] |  +- antlr:antlr:jar:2.7.6:compile
[INFO] |  +- cglib:cglib:jar:2.1_3:compile
[INFO] |  +- asm:asm:jar:1.5.3:compile
[INFO] |  \- commons-collections:commons-collections:jar:2.1.1:compile
[INFO] \- junit:junit:jar:3.8.1:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------

From this output, we can see that the persistence-api dependency is coming from hibernate. A cursory scan of the source in this module will reveal many javax.persistence import statements confirming that we are, indeed, directly referencing this dependency. The simple fix is to add a direct reference to the dependency. In this example, we put the dependency version in simple-parent’s dependencyManagement section because the dependency is linked to Hibernate, and the Hibernate version is declared here. Eventually you are going to want to upgrade your project’s version of Hibernate. Listing the persistence-api dependency version near the Hibernate dependency version will make it more obvious later when your team modifies the parent POM to upgrade the Hibernate version.

If you look at the dependency:analyze output from the simple-web module, you will see that we also need to add a direct reference to the simple-model dependency. The code in simple-webapp directly references the model objects in simple-model, and the simple-model is exposed to simple-webapp as a transitive dependency via simple-persist. Since this is a sibling dependency that shares both the version and groupId, the dependency can be defined in simple-webapp’s pom.xml using the ${project.groupId} and ${project.version}.

How did the Maven Dependency plugin uncover these issues? How does dependency:analyze know which classes and dependencies are directly referenced by your project’s bytecode? The Dependency plugin uses the ObjectWeb ASM (http://asm.objectweb.org/) toolkit to analyze the raw bytecode. The Dependency plugin uses ASM to walk through all the classes in the current project, and it builds a list of every other class referenced. It then walks through all the dependencies, direct and transitive, and marks off the classes discovered in the direct dependencies. Any classes not located in the direct dependencies are discovered in the transitive dependencies, and the list of “used, undeclared dependencies” is produced.

In contrast, the list of unused, declared dependencies is a little trickier to validate, and less useful than the “used, undeclared dependencies.” For one, some dependencies are used only at runtime or for tests, and they won’t be found in the bytecode. These are pretty obvious when you see them in the output; for example, JUnit appears in this list, but this is expected because it is used only for unit tests. You’ll also notice that the Velocity and Servlet API dependencies are listed in this list for the simple-web module. This is also expected because, although the project doesn’t have any direct references to the classes of these artifacts, they are still essential during runtime.

Be careful when removing any unused, declared dependencies unless you have very good test coverage, or you might introduce a runtime error. A more sinister issue pops up with bytecode optimization. For example, it is legal for a compiler to substitute the value of a constant and optimize away the reference. Removing this dependency will cause the compile to fail, yet the tool shows it as unused. Future versions of the Maven Dependency plugin will provide better techniques for detecting and/or ignoring these types of issues.

You should use the dependency:analyze tool periodically to detect these common errors in your projects. It can be configured to fail the build if certain conditions are found, and it is also available as a report.