I’ve seen my fair share of projects migrating from Ant to Maven, and, for a complex project, this migration path can take some time. You have to worry about dependency management, project structure, and retraining an existing team to use Maven and understand the core concepts behind the tool. When you make the shift, you are often affecting development infrastructure for an existing project, and you need to take into account development environments as well as developer’s ideas about how code should be organized and stored in source control. In this post, I’m going to discuss a common pattern I’ve seen in Ant to Maven migrations: how to migrate the monolithic project.

A Common Pattern: The Monolithic Project

Ant projects which have evolved over many years often lack modular structure. While it is certainly possible to create the equivalent of a multi-module Maven project in Ant, the usual progression in an Ant project is to store all of your source in a single tree and use extra targets to selectively compile different packages. This approach is shown in the following figure.

In the most extreme cases, there is a single project which contains an array of modules. Maybe you have an entire enterprise system all stored in a single project alongside a complex Ant build.xml file which contains a collection of tasks for each component. A monolithic Ant project usually produces a series of artifacts: a JAR containing some API for clients, a server-side web application, a utility library, etc. Developers have usually grown so accustomed to this approach that the idea of splitting up a complex system into a series of related submodules can see daunting.

Moving to Maven: Do we need all these projects?

One of the first questions I get in a Maven migration from Ant is whether it is really necessary to modularize a project. “Do we really need to create all these projects for Maven?” (Short answer: Yes, there is no avoiding this.) This is usually accompanied by a concern that Maven limits each project to producing a single artifact. These are two core assumptions of Maven: your projects will be modularized and each project should produce a single build artifact.

Now, while these are core assumptions, there are ways around them, and I’ve even seen people go to extreme lengths in an attempt to preserve this single, monolithic project. You can create an elaborate set of profiles to modify the build depending on the context in which it is run, and you can attach extra build artifacts to a project using assemblies. Even though Maven makes assumptions about code layout and project structure, it can do just about anything, and in this case, it can be used to approximate the monolithic, combined Ant project.

If you do this, if you try to trick Maven the first thing you’ll notice is that your monolithic project’s POM is going to be somewhat unwieldy: it is going to be massive. Instead of a simple, declarative picture of a project, you are going to have a POM that contains multiple assembly definitions. Each assembly definition is going to have to explicitly define the structure of your build artifacts and you are going to find yourself venturing into includes and excludes patterns for things like the Maven Compiler plugin. In other words, you are going to have to expend a huge amount of effort to bend Maven to your assumptions.

If you do this, you really won’t be using “Maven”. You’ll be using your own interpretation of Maven, and once you go down this road, you are going to start having problems using standard tools designed for Maven. In short, don’t do this. Don’t try to bypass Maven’s approach to modularity with assemblies and profiles. If you do, you are going to find yourself “swimming upstream”, and your first hint is going to be the size and complexity of your POM files. I’ve certainly created some complex POMs in my time, but I only need to do this for projects that really require customization because they are doing something unique (see the POMs for the Maven book builds, they are large and extremely customized).

If you are creating web applications, EARs, and simple Java applications you shouldn’t have large POMs. If you do, it is probably a sign that you have ventured too far “off the map”. While you might be building your project with “Maven”, there is likely something wrong with your approach. If you find yourself constantly challenging an assumption as basic as project modularity, then you need to either rethink using Maven entirely or rethink your project structure.

Adopting Maven Means Adopting Modular Structure

If you are going to adopt Maven, you must adopt a modular project structure. If you don’t you will be fighting an uphill battle with Maven and the tools that have been designed to work with it. You will be fighting not only with Maven, but you’ll be doing constant battle with your IDE and your repository manager.

If you are moving from Ant to Maven, do you yourself a favor. Adopt a modular project structure. Don’t approach Maven as a “toolbox” like Ant with tasks and targets. Approach Maven as a framework with expectations and assumptions. If you do this, you’ll have a much easier time adopting the tool.

Sonatype is pleased to announce version 1.5 of the Nexus Migration Plugin. This version contains an updated implementation of our Artifactory to Nexus migration tool. If you are currently running Artifactory and you want to migrate to Nexus, all you need to do is:

1. Create a System Export from Artifactory
2. Install Nexus with the Nexus Migration Plugin
3. Import Your Artifactory System Export to Nexus
4. Configure the Artifactory Bridge to seamlessly serve artifacts to existing clients

You can use the Nexus Migration Plugin with Nexus Open Source or Nexus Professional. To get started, download Nexus Open Source or Nexus Professional, then download the Nexus Migration Plugin. Read Appendix A of the Nexus Book and learn how to:

• Install the Nexus Migration Plugin
• Create a System Export in Artifactory
• Import an Artifactory System Backup into Nexus
• Configure Repository Mapping and Import Settings
• Map Legacy Artifactory URLs to Nexus

Sonatype is pleased to announce the availability of our most popular book, Maven: The Complete Reference, as an epub. Click here to download the book as an epub and start reading this book on a mobile device.

If you have any questions or feedback, we encourage you to send email to book@sonatype.com. If you have any problems to report, please include the section number and the type of mobile device you are reading this book on. This book has been tested on iPad, iPhone, Sony Reader library and the Adobe Digital Edition reader.

If you are new to Maven, don’t forget to check out Nexus Open Source and Nexus Professional. If you develop with Maven, you should be using Nexus. Once you start using a Repository Manager, you won’t believe you ever developed without one.

We’ve received a steady stream of requests to release “Maven by Example” and “Maven: The Complete Reference” in the ePub format. Today, we’re announcing the availability of Maven by Example as an ePub. Download the Maven by Example ePub today and start learning about Maven.

This release has been tested on Apple’s iBook on both an iPad and iPhone and in the Adobe Digital Editions reader. To read this book on an iPad or iPhone: install the latest version of iTunes, download the ePub from Sonatype, and import the file using “Add to Library…” from the File menu.

We would appreciate any feedback you might have about formatting and layout. If you have questions, comments, or problem reports about our ePub books, send an email to book@sonatype.com. If you are reporting a typo or a problem with our ePub books, please make sure to include the device and software you are using as well as the section number, as page counts can vary depending on text size and device layout.

Stay tuned for “Maven: The Complete Reference” in an ePub format.

In the previous posts in this series, we’ve talked about how a repository manger changes the development cycle. In this post, I’m going to talk about how a repository manager can be used as a way to interact with third parties. Specifically, I’m going to talk about vendors and partners.

While Maven Central contains a huge number of libraries, there are still some pretty important things missing. Anything covered by a non-open-source license, like the Oracle JDBC drivers or anything proprietary, isn’t going to be made available from the Maven Central repository. Very often there are going to be commercial libraries that you will need to download and install in a third party repository. While this works, it requires some manual work to upload the artifact.

Vendors and Repository Management

When your vendors embrace repository management, there’s no reason why you would have to manually install any third party JARs in your corporate repository. Instead, vendors who need to provide binaries to customers would provide them via authenticated public repositories. These vendor-run repositories, protected by authentication credentials, would allow customers to synchronize local, corporate repository managers with commercial software vendors. This would allow vendors to capitalize on a new, “always connected” model for software delivery.

Now, the situation for the customer is simple. Once they set up an authenticated proxy repository to connect to a vendor’s repository, the vendor can deliver new updates and software directly to a customer’s repository.

The situation for the vendor is equally straightforward.  Instead of having to send the customer a packaged set of binaries, they simply tell the customer to update a version number in a dependency. The build will then interrogate the customer’s repository and then automatically download the repository from the vendor.

Both the customer and the vendor have fewer moving parts to worry about, and the task of delivering software becomes very simple.

The vendor receives another benefit from this setup. They are now able to keep track of which customers had requested which artifacts. The vendor can control which artifacts or features of a product suite a customer has access to, and they could use the repository manager as a way to enforce licensing or provide metered, on-demand access to software components.

Partners and Repository Management

In today’s increasingly connected world, it is very common for one company to provide an API for partners. Sites like Twitter and Google provide APIs for the general public, and other companies provide partner-specific APIs for collaboration. If you provide an API for a partner company, you can expose libraries and interfaces using a public-facing, authenticated repository.

Suppose that you work at a financial institution that needs to partner with another financial institution. The two institutions have agreed to collaborate with one another using a simple set of REST services. While these REST services are very well documented, you also want to make sure that your partner is invoking these REST services according to a set of standards. To make it easier to consume these services, you’ve created a simple client library for your partner.

To make it even easier for your partners to use this library, you’ve published versions of this library to a public-facing, authenticated repository. Using the security features of a modern repository manager, you can create partner-specific library versions and limit access to just the intended recipients. You can also maintain detailed audits of when a particular API was delivered to a particular partner.

As more and more organizations start to adopt the best practice of running a repository manager, these same organizations will be creating ad-hoc, private relationships to emulate the ease of Maven Central for a corporate environment. Partners, vendors, and consortia of companies will create repositories to facilitate code sharing and cooperation.

In the last post of this series we talked about how a repository manager, when coupled with a continuous integration server, allows you to work on individual modules in complex multi-module projects. In the post before that, we talked about the repository as something that enables greater collaboration between workgroups. In this post, I explore how using a repository manager simplifies deployment.

The following diagram shows the process of pushing code to production and automating a deployment:

This post will first discuss production deployments by setting up a hypothetical example. Assume that your organization runs a fairly complex application that consists of a few back office systems, a CRM system, and a web application. When your company needs to deploy one of these systems to production, the result is a highly orchestrated symphony of activity. Servers need to be taken offline in a coordinated manner across a distributed network of machines. Downtime needs to be planned and communicated to the right individuals, and backups need to be created before any new code is pushed to a production system.

In other words, any time you need to push new code to production, it is a real challenge to make sure that everyone is coordinating in just the right way. A few days before a production deployment, the development team freezes a particular branch and tags a release. This release is then installed in a QA environment. The QA team tests the release candidate, and if all goes well, this release is then deployed to production.

Without a repository manager, your operations teams has to know how to checkout a release tag from SCM, they have to run the entire build to generate a few binary artifacts, and then they need to write some scripts to automate a deployment to production.

With a repository manager, you can publish a release candidate binary to a hosted repository. This release candidate binary can then be used by a series of deployment scripts (or a tool like ControlTier or Puppet). You can have a deployment script that will publish the deployment to a QA environment, and then you can have a deployment script that takes the same, certified binary and publishes it to production.

The main difference between these two approaches is that the second approach adds some isolation between the SCM system and the deployment script. If your production deployment depends directly on your build, you have to recompile and repackage your entire system every time you do a deployment. This is often a challenge because the people that run the build system for deployment are seldom the same engineers that develop your system. If your operations team has to check out source code from an SCM just to build a production network, you are introducing some risk into your process.

Instead, use the repository manager as a way to share binaries with operations. If you are deploying applications to QA and production, having to rebuild an entire system from SCM just to deploy a system to an environment is an unnecessary risk. Instead of certifying tags in an SCM, your QA testers should be certifying binaries. This way you can be certain that the systems you are deploying to production have been tested.

In the previous post in this series I discussed three compelling ways in which a repository manager can benefit the development cycle. It proxies artifacts locally, it is optimized to store binary artifacts, and it facilitates a new level of collaboration and agility that isn’t possible when your SCM is only way for workgroups to collaborate. In this post, I’m going to talk about how a repository manager works in concert with a continuous integration server like Hudson or Bamboo.

First, the how, what, and when of a continuous integration server. Continuous integration (CI) servers are an established fact of of modern development infrastructure. It is a server which, for the most part, waits and watches. It keeps a vigilant eye on your source control system and jumps into action every time it sees a code change. When code changes, your CI system is usually configured to run the entire build, execute all of your unit and integration tests, and send out an email to every developer if it identifies a defect or a failed test.

It does this so that you will have an easier time identifying where a particular problem was introduced to the source code. If John checks in some bad code, the CI system runs the build immediately, and about 30 minutes later, everyone in the group receives an email with the subject header “John just broke the build”. It is a great way to identify errors, and it is also a great way to motivate developers to test locally before committing to a source control system as no one likes to be the reason for a build failure email.

Running a CI server is more than “just a good idea”. Once your system reaches a certain level of complexity you can’t scale a system without commiting to continuous integration and testing. If you don’t have continuous integration, you end up having to put all development on hold each time you want to perform a release. If you don’t build, test, and deploy your system on a regular basis – if it isn’t something that is well rehearsed, integration becomes a time consuming nightmare of manual testing and builds that often leads to inconsistent builds. This is especially true if your development effort spans multiple systems and multiple development workgroups. You run a CI system because building, testing, and deploying your system should be automatic: it should be as trivial as pressing a button.

The concept of a CI server is only slightly more established than a repository manager, and very often you will see that an organization has identified the need for a CI server before they’ve identified the need for a repository manager. If you are coding a complex system, there is a very good chance that you are already running a CI server. The most popular servers out there are Hudson, Bamboo, and CruiseControl. While the connection between CI servers and repository managers might not be immediately obvious, when used together they can introduce some new possibilities for the way you develop your systems.

Continuous Publishing

When you have a system to continuously build your code, you also have a system that can continuously publish SNAPSHOT artifacts to a repository manager to enable a more granular approach to development. What do I mean by “a more granular approach to development”? To answer that question, let’s take a look at a complex multi-module project using the example of the eCommerce group from the previous post in this series.

Assume you have a new programmer starting tomorrow. Instead of throwing him at the entire forty-thousand lines of code, you would like to be able to give that developer a small, easy to digest task. You want this developer to add support for PayPal’s Adaptive Payments API in your eCommerce system. That’s it. You don’t want them to be distracted by the overwhelming scope of the project, and you certainly can’t afford for them to take a three month voyage through your project’s code before they start contributing to the effort. Deadlines are tight, and you don’t have enough people on your team. It is important that new hires start programming as soon as they walk in the door.

Without a repository manager hooked up to a continuous integration server, if you try to checkout just the ecom-paypal project, the build is going to fail because it will try to download dependencies from a repository manager. In the case of the ecom-paypal project, assume that the dependency graph looks like this.

When you have a repository manager and a continuous integration server, you can configure your continuous integration server to publish SNAPSHOT artifacts (in-progress SNAPSHOT binaries) to your repository manager. This will allow you to just check out a single, isolated portion of a much larger multi-module project.

Without a repository manager, trying to build version 1.3-SNAPSHOT of ecom-paypal in isolation is going to generate errors because you are forced to checkout the entire codebase to build and install all of the dependencies in your local repository. With a repository manager, SNAPSHOT artifacts are being continuously published because Hudson is checking you SCM every few minutes and building the latest code. When you run the ecom-paypal module’s build in isolation, Maven is going to download the most recent SNAPSHOT.

Without a repository manager, your new developer is going to have to download the entire codebase and run a large time-consuming build. With a repository manager you can work on specific components of a larger multi-module project. This ability to divide and conquer your codebase comes in very handy when you need a consultant to take a look at a specific problem, or when you need to look at a coding problem in isolation.

When you continuously publish build artifacts to a repository manager, you move away from the single monolithic project build and toward a project layout and architecture that lends itself to modularization.

In tomorrow’s post: How a Repository Manager decouples deployments from source code, and what that means for developer operations.

In the first part of this series, I gave you a glimpse at the big picture of repository management, and I listed some common anti-patterns present in systems that haven’t yet installed a repository manager. In this post, I’m going to focused on the benefits a repository manager has on the development cycle. How does using a repository manager change the way your developers will approach development? What problems does it solve? And what possibilities does it introduce?

Caching artifacts locally

The first, and most obvious, benefit is that a repository manager will proxy and cache artifacts from a remote repository. If you use Maven or another tool that can download artifacts from Maven Central, you have probably had the experience of checking out a large build from source control and running it, only to have to wait for minutes while the build downloads dependencies from Maven Central.  An even bigger frustration than that, if your connection to the internet is tenuous, or if Central happens to be unavailable for whatever reason, you cannot build your code.

With a repository manager, downloading your dependencies takes much less time and your builds don’t rely on internet access. Unless this is the first time building a particular project, all of the artifacts your build uses are going to be cached at the repository manager. Instead of waiting for artifacts to be downloaded over the public internet, your build will download artifacts from a local server. Since you avoid the public internet, this process is sped up significantly. The build that once took 15 minutes to download dependencies will now take less than a minute to download everything it needs.

Get those JARs out of Subversion

The Oracle JDBC driver, a proprietary JAR from a vendor, is the kind of one-off, third party library that is not going to made available on a public repository. Without a repository, the most obvious solution is to just check these JARs into source control and store them right next to your code. When you run your build, branch your project, and release your code you are just passing around these binaries as a part of your project. While this approach does work, you are overloading the source control system to perform a task it wasn’t designed for.

While Subversion, and many other modern SCM tools, will gladly version binaries, you are storing a static, unchanging artifact in source control. Every time you checkout out source control, you are checking out binaries, and if you are unlucky enough to work somewhere that stores all libraries in source control, you will often be asked to check out a large repository of JARs. Your SCM is going to carefully keep track of changes and version files that will never change.

This is both inefficient and unnecessary. If you use a repository manager, you can upload third party libraries to a third party repository that stores artifacts on the repository manager. Then you can mix these third party libraries into a public repository group that will combine the contents of public repositories with your own repositories. In other words, you can declare a dependency on a custom, manually uploaded third party JAR as easily as you would declare a dependency on a library stored in Maven Central. Don’t store binaries in source control, use a repository manager instead.

Collaboration

Collaboration is best illustrated with a hypothetical organization. Imagine you work at an organization with three large development groups of about 30 developers. There is an eCommerce group, which is responsible for writing systems that interact with banks and service providers like PayPal. There is a CRM group, which has the unenviable job of merging two disparate CRM solutions from Peoplesoft and Oracle and wrapping those systems in an easy to use Java API. Then there is a third group, the web applications group, which has the primary responsibility for wrapping these two back office systems in a slick web interface. These relationships are captured in the following diagram:

Imagine that the eCommerce group and the CRM group have to assemble the entire external API into a single project: ecommerce-api and crm-api. These APIs contain all of the logic required to connect to a set of internal services hosted by each group. As the eCommerce group and CRM groups innovate and offer new services, they will cut releases of these new APIs and publish those releases to the repository manager.

When the web applications group is ready to start using a new version of the eCommerce or CRM API they can simply define a dependency on a version of this artifact. The build for the web applications group will then download the appropriate version of the eCommerce of CRM API from the corporate repository manager.

In this way, the repository manager is a central collaboration point for difference groups within the same company. Without the use a repository manager, you will often see disparate groups forced to check each others code out of SCM and build it from scratch just to generate client libraries. In the organization that has embraced repository management, these libraries are published as binaries for clients (in this case other workgroups) to consume from the repository manager.

When you collaborate using the SCM, it is very difficult to scale. Every workgroup is forced to use the same set of tools for build management and you will very often see the entire IT operation having to synchronize releases. When you share build artifacts using the repository manager, you decouple workgroups from one another and allow your internal teams to collaborate using a more adaptive, open-source model.

If the eCommerce group needs to innovate, they can do so without affecting the source code of the web applications group. They can publish new releases to the repository manager, and the web applications group can decide when and how they are going to consume these releases by changing the version of a dependency in a build.

When you collaborate using the SCM, all of your workgroups are joined at the hip by a common codebase. When you use a repository manager you allow different workgroups to innovate and create at their own pace.

In next week’s posts: Continuous Deployment and Integration

Whenever I speak to someone doing Java development, I always ask if they are using a repository manager. Repository managers are still an emerging technology, but I’ve noticed a consistent trend: more and more developers view a repository manager as an essential part of development infrastructure. This certainly wasn’t the case just two years ago, and I think that the big motivator behind this trend is that the quality and stability of Maven Central has improved remarkably because of the efforts of people like Brian Fox and others who are focused on making the service more stable.

Another reason why we’ve seen more adoption is that most developers understand the benefits of using a tool like Maven for automatic dependency management. In 2005, it was common to see projects store binary JARs alongside source code in projects. In 2010, you rely on the repository and the metadata it contains. If you use a library like Guice, you’ll add a dependency on the artifact and let your build tool take care of the details. To do otherwise would be to commit yourself to a manual work updating JARs and testing dependencies each time a new version of an external library is released.

Despite the increasing prevalence of repository managers, I still stumble upon workgroups and organizations that haven’t heard of repository management. When you ask if they are using a repository manager, they might think you are referring to Subversion or source control. This series of posts is a high-level overview of the main benefits of repository management. If you are trying to convince someone to start using a repository manager, the next few blog posts are for you.

Repository Management: The Big Picture

Compare the diagram shown above with the diagram shown below. In the next few posts, I am going to emphasize the specific benefits of using a repository manager. Specifically, I’m going to talk about:

• How a repository manager changes the development cycle
• How continuous integration is used to continuously publish internal build artifacts
• How a repository manager simplifies the process of building and deploying systems to production
• How a repository manager can act as a gateway between vendors and external partners

When you don’t use a Repository Manager

Before I get started on the benefits of repository management, I want to talk about the realities you face when you don’t use a repository manager. Here are some common anti-patterns when you don’t use a repository manager:

• All of your developers download artifacts directly from public repositories. A new developer starts on a Monday. That developer will spend an hour downloading a massive library of dependencies from Maven Central. Worse, if Maven Central happens to be down that day, they will be out of luck entirely.
• Proprietary or Vendors libraries are passed around, from developer to developer. If you don’t use a repository manager, how do you distribute the Oracle JDBC driver? Maybe you place it in a shared file system and tell people to download it and install it in ~/.m2/repository. More likely, developers just pass this JAR around as an email attachment with some ad-hoc instructions.
• JARs are checked into source control. If you don’t use a tool like Maven, which knows how to download artifacts from a remote repository, you might be following the very common pattern of checking binary dependencies and libraries into source control. I’ve seen many instances of companies creating ad-hoc JAR repositories and checking these repositories into source control, only to version and branch these static binary files with every release.
• The source control repository is used to store everything from source code to binary builds. Because there is no repository designed to store binaries, developers start to use tools like Subversion to keep track of binaries. As time passes, the Subversion repository becomes an ad-hoc file system for files that have no business in an SCM.
• The continuous integration server depends on public repositories. When you change your build or add a new dependency, your CI system downloads dependencies from the public repo. It depends on the availability of this public resource to run builds.
• Production deployments have to run the entire build, from start to finish, to generate binaries for deployment. When a build is tested and then ultimately pushed to production, the build and deployment scripts checkout source code, run the build, and deploy the resulting binaries to production systems.
• Sharing source code with external partners means granting them access to your SCM.  Since there is no established mechanism for publishing source or binary artifacts, the only way to share code with partners is to either send an archive of your source, or provide them with direct access to your SCM.

The general theme in all of these anti-patterns is that either your systems depend on public resources, or they all depend on the SCM system as a central collaboration point. In the next few posts, I’m going to detail how using a repository manager provides a solution for each of these issues. I’ll go into why each of these anti-patterns is a bad idea, and how you can use Maven, Nexus, and Hudson together to solve these problems and create a more efficient software development effort.

Stay tuned for the next post: Caching and Collaborating.

Craig Tataryn of BasementCoders.com interviewed Brian Fox about the upcoming release of Maven 3. From Basementcoders.com:

“With the Maven 3 betas being out and packing a tonne of cool features we decided that we needed to sit down and talk with someone in the know. Enter Brian Fox, PMC Chair Apache Maven and VP of Engineering at Sonatype (the fellows who make Maven Repositories manageable via their Nexus product).”

This interview touches upon Brian’s background in build management and developer infrastructure, and how Brian started to use Maven and became involved in the Apache Maven project. You will also hear Brian discuss the history of Maven, the motivations behind the project, some comparisons between Maven and Ant, and new features planned for the Maven 3 release.

To listen to the full interview, click here.