You know who this guy is? Probably not, he’s Rube Goldberg.

I’m surprised by how few engineers know his work. Rube Goldberg was a cartoonist who lived from 1883-1970, he’s famous for drawing cartoons of ridiculous and inconceivably complex machines. His work was important during a time in which the world was becoming increasingly mechanized and automated providing a sort of cultural “steam vent” – a way for people to poke fun at machines and industry. I’d embed his work here, but none of it is public domain, so see for yourself or search Google Images. (Be warned, you can spend hours looking at these cartoons.)

I learned about Rube Goldberg from an Engineering professor who, at the time, said, “Rube Goldberg is the most important thing you’ll learn over the next four years”. Back then, we all thought he was joking, but it turns out that he wasn’t. In fact, I wish more people, especially “build engineers” had some exposure to these cartoons. If they had, they’d take a step back and realize that there has to be a better way.

“accomplishing by complex means what seemingly could be done simply”

The title of this section is the Webster’s New Dictionary’s definition of “Rube Goldberg”. Does this describe your internal software distribution process? I fear it might. I’ve seen some builds and deployment procedures that convince me that people are actively trying to create new Rube Goldberg machines with SCM tools and build scripts. Almost every month I see some new variation of what I eventually characterize as a “Crazy Rube Goldberg” approach to releasing software. Systems that barely work, and that no one dares to change for fear of breaking some crazy contraption.

Goldberg’s cartoons and the build’s inspired by them share common characteristics. The complexity of his cartoons are almost always defined by the complex ways in which some object is transported from one “stage” of the cartoon to the next and the comic “indirection” involved in every process.

A cartoon example, a man slips and falls, stepping on a switch which then activates a mechanical hand that throws a bag of sand on to a scale which then triggers a mechanism to fire an arrow at a balloon. A build example, Jeff runs a build on his laptop because that’s the only place release builds work properly, he then checks the WAR into Subversion and then sends a series of emails to several other people. A deployment engineer then takes the output of Jeff’s build and runs it through a custom, somewhat confusing Perl script to customize a series of configuration variables.

Common Characteristics

The common characteristics between Rube Goldberg cartoons and Rube Goldberg builds are:

• Unnecessary Indirection

  • Cartoon: Instead of just creating a machine to pop a balloon, a cartoon creates a contrived process that involves unnecessary (and often impractical) indirection involving pulleys. Goldberg’s cartons often find inventive ways to convert potential energy to kinetic energy in some fantastic way you wouldn’t have thought of.
  • Awful Build: Instead of using a technology to keep track of binary artifacts, a Rube Goldberg build overloads SCM or (worse) email to send multimegabyte binaries around. Instead of using a technology with built in support for automatic email notification, your build relies on some ad-hoc system of people remember to send just the right individuals a set of process-related emails.

• Reliance on the Unlikely

  • Cartoon: A bag of sand thrown across a room lands exactly on a bulls eye of a target triggering yet another unlikely series of events, continue, repeat…
  • Awful Build: A complex build that takes more than an hour to complete has a random dependency on a web service that provides an series of configuration variables used to configure a binary for production. If the web service happens to be unavailable, or if it returns invalid results, the entire build process must be started from scratch.

• Reliance on Animals (or Humans)

  • Cartoon: One of the comic elements almost always present in a Goldberg cartoon is the presence of an animal of some sort. A bird who, when hit with an arrow squawks loud enough to scare a dog into running and subsequently lifting a weight via a pulley system.
  • Awful Build: Awful build systems frequently require direct human intervention to the point where the build doesn’t work unless the same individual is always “strapped” into the build in just the right way. “After that compilation step completes, Jack usually logs in and modifies the build number before he runs the deployment script”.

• Comedy

  • Cartoons: Rube Goldberg’s cartoons are comedic because they involve the impractical or the impossible. Readers find them amusing because they are plausible, yet inadvisable ways to implement a process.
  • Rube Goldberg-style builds are comedic because they involve the impractical or the impossible. Developers find them amusing because they are plausible, yet inadvisable ways to implement a process.

Build/Deploy systems are Comedy?

If they work, they can be. It’s quite common for the build/deploy lifecycle at a corporation to involve a series of steps that no one fully comprehends. Maybe there’s a crazy 4000 line Ant script or maybe there’s a multi-headed Hydra of bash magic that happens to produce a working system. In the last few months I’ve talked to people with build systems that work, but no one on staff understands how. When you ask about these release procedures people often chuckle a bit – “Oh, right, our release process…. well let’s say it’s a little confusing…”.

“Why does the build do it like this?” is often met by, “I don’t know, but I’m too scared to change this thing right now, the contractor left us with this and it works… for now.” In other words, many of these builds involve a series of unknown pulleys and unexpected indirections that resemble a cartoon.

In the next part of this series, we’re going to talk about how Nexus can be used to untangle a build into something more reasonable. Something that doesn’t involve pulleys, magic, and a series of unlikely events.

Note: Rube Goldberg doesn’t do it for you? Then just watch the OK Go video “This Too Shall Pass” to get a modern idea of what I’m talking about. If you haven’t seen that, you should. In fact, if you haven’t seen this video yet, stop reading our boring Nexus blog and take a break, it’s worth it.

Over the course of the past few years, I’ve interacted with hundreds of people when talking about build tools and repository management.   It continues to surprise me how many people don’t realize where these artifacts come from.   When you run a build and these JARs just show up alongside all of their dependencies, it’s like magic to most people.     If you know how it works, it’s very obvious to you that running a repository manager is the right thing to do.  This post is a reminder to everyone using build tools that rely on Central: take time to proxy Central with a repository manager.

“Wait, that’s how Central works?”

There’s something so automatic about dependency management in Maven that it often takes people a few months to understand exactly where those JAR files are coming from. In an 8 hour Maven class, I get to dependencies in the third hour, and after describing Central, what it is was like before Central, how metadata is stored in a repository alongside binaries, transitive dependencies, etc…. it all falls into place, and people realize that this simple thing they’ve grown accustomed to is only easy because of a ten year effort to refine the model, the creation of a support structure for source forges at places like Oracle and Google, and a constant investment in infrastructure.

On the one hand, it’s a great success that Central is, for the most part, an invisible utility that supports developers.  On the other hand, it’s the kind of thing that people can start to take for granted very easily.

For example, a few months ago I spoke to someone who worked in an environment disconnected from the internet for security reasons.   This individual was talking about how limiting it was to have to download JARs from open source projects manually and assemble them in a project.   His words were: “It’s like programming Java in 2001 again.”

How can you help?

Imagine millions of developer spread all over the world: different time zones, different applications, but they all hit the same service: Central.    Some regions have more developers than others so we certainly see peaks in usage throughout the day, but in general, Central’s serving thousands of files throughout the world at any given time during the day.

Maybe someone just installed Maven for the first time, or maybe they blew away a local repository, with numbers like these we see a world that has a constant appetite for artifacts.   It isn’t a problem for Central, and I’m not writing this because Central is falling down on the job. Central can handle it, but it certainly isn’t the most efficient way to support millions developers.  It isn’t a good use of network bandwidth, and it isn’t a good use of energy to constantly cart around the same static JARs over and over and over again when the solution is so easy.

If everyone who used a build tool that interacted with Central adopted a repository manager such as Nexus we’d have a faster, more responsive system.   Central’s maintainers would be focused less on addressing the occasional runaway build and could spend more time and resource on increasing availability and functionality of this essential service.

Broken Builds

The other factor playing into this is that Maven builds only download releases once.   It isn’t like these build tools are repeatedly returning to Central to download release artifacts over and over again.

Well… actually… that isn’t true, we’ve seen some installations of Hudson configured to delete a local repository before every build placing a high load on Central.   Imagine a build that downloads 50 MB of dependencies running once every 5 minutes.  That’s one build consuming ~14 GB a day never mind the time wasted downloading static artifacts. While these broken builds are the exception, they do still show up from time to time.  Central can handle the load, but imagine 1000 of these broken builds running continuously and you can see the challenge.

A Simple Reminder: Please Proxy Central

We’re constantly watching the performance of the system and making sure it stays up and running for an entire world of developers.  If you use a build tool that hits Central whether it is Buildr or Maven or Gradle or Ivy, you can help us by running a Nexus instance.

Even if all of your builds work perfect, running a local Nexus instance helps preserve Central as a public, free resource and it will lead to faster, more responsive builds.

Sonatype is going through the archives and digging up articles that we think would be useful to developers using our tools. If you use Maven, keep reading the post below from Sonatype Vice President of Engineering Brian Fox on Maven best practices and how-tos.

We have a handful of Maven best practice and how-tos documented in the blogs. Over time they get buried by newer posts, but the content is still just as relevant. Learn more about Maven by checking out the following blogs:

(1) Why Putting Repositories in your pom is a bad idea

How to properly manage repository definitions and why you shouldn’t declare them in your poms.

(2) Best Practices for Releasing with 3rd Party Snapshot Dependencies

How to most effectively create your own internal release of a snapshot dependency.

(3) Maven Continuous Integration Best Practices

Multiple techniques for running Maven builds from a CI system.

(4) How to Detect if you have a Snapshot Version

See the code that Maven uses to detect a snapshot version. This is helpful if you have non-standard release versions to make sure they won’t be detected by Maven as a snapshot.

(5) Optimal Maven Plugin Configuration

Tips on how to configure plugins in a way that makes it easier to understand what is happening in your pom.

(6) Adding additional source folders to your Maven build

How to use the buildhelper plugin to add additional source folders to your build. This is common if you have tools that generate sources that don’t have a Maven plugin to add the source folder automatically.

(7) Misused Maven Terms Defined

Find out about several terms that are often misquoted. This post also specifically discusses classifiers and attached artifacts.

(8) How to override a plugin’s dependency

How to override a plugin dependency in your own pom…for example if you want to use a newer version of a tool wrapped by a plugin (pmd, checkstyle, cobertura are common examples)

(9) How to share resources across projects

How to share resources in a repeatable way without using relative paths in your poms. It also shows a technique for sharing checkstyle and pmd configurations across all your modules.

(10) How Plugin versions are determined

Learn about how Maven determines which version of a plugin to use, and why you should use PluginManagement to control the versions yourself.

(11) How to convert from Ant to Maven in 5 minutes

Okay so it might take longer than 5 minutes to convert from Ant to Maven, but this post will show you that it is possible to work the migration in parallel so that most of your team sees the conversion happen very quickly when the time is right.

(12) How to make an executable JAR

Step-by-step instructions on how to make a standalone JAR (i.e. one with no dependencies) executable.

Note: If you have suggestions for further topics, please leave a comment with your ideas.

Some questions about syncing Maven repositories between two sites were recently asked on GetSatisfaction.com.

“We will be moving data centers and want to setup another Maven2 repo that is managed with Nexus OSS. We want both repositories to be online and read\writeable until we migrate all our environments to the new site. I have a few questions:

1. What is the best method for copying the repo to the new location?
2. What is the best method for keeping the two repos in sync? We want to minimize network bandwidth usage.”

The video below answers these questions, and offers multiple solutions:

We are offering one last virtual Maven Mechanics (MVN-101) training course for European business hours in June. This will be our last virtual class in CEST for the summer. Be sure to enroll today!

Course details:

• Course: Maven Mechanics (MVN-101)
• Date: Tuesday June 14, 2011
• Time: 9:00AM – 4:00PM CEST (UTC/GMT + 0200) *European Hours
• Enrollment Fee: $595 USD (~425 EUR)

ENROLL TODAY!

Maven Mechanics (MVN-101) is the ideal course for programmers who work with Apache Maven projects and need to understand how to work with an existing Maven build. If your team is just delving into Maven and Nexus, this class is the easiest way to make sure that everyone starts from the same foundation. This class is also appropriate for the existing Maven user who is interested in developing a greater understanding of the Maven fundamentals. Learn more here.

You will leave this Maven tutorial equipped with a full understanding of the Maven Project Object Model (POM) and a firm grasp of the underlying fundamentals of this development kit including:

• The Maven lifecycle
• Maven plugins and goals
• Multi-module Maven projects
• The contents of the Project Object Model (POM)

The Maven 3: The Future of Enterprise Java Build Infrastructure presentation is now available for viewing. This presentation was given at EclipseCon 2011 by Sonatype founder Jason van Zyl.

More on this presentation:

Maven 3 is the best version of Maven yet. Maven 3 is faster, has been optimized for IDE use, and is fully backward compatible with Maven 2. One of the big focuses of Maven 3 is to provide a more reliable, more stable and better performing build tool. Faster Maven builds lead to higher developer productivity in your organization.

Watch the video below:

One important difference between Hudson Open Source and Hudson Professional is how they support Maven 3.

Sonatype has developed state-of-the-art support for Maven 3 focusing on creating seamless integration between the internals of Hudson CI and Maven.   Year of work on Maven 3 internals to support more efficient embedding along with our multi-year investment in bringing both JSR-330 support and GWT UI integration to Hudson CI  has produced a CI system that support Maven 3 like no other on the market.

If you’ve used Hudson and Maven together in the past, you might be familiar with the Maven 2 project type that provides users with the ability to define a Maven build that is aware of a project’s POM. While we understood the motivation for a Maven-specific project type, we took our Maven support in an entirely difference direction. We created a Maven build that can be used a part of a larger freestyle build, and the reason we did this was to get away for the idea that a Maven build can only include one step, one call to a series of goals.

Sonatype found this to be more of a limitation than an advantage.

Supporting Real-world Maven Builds

Why?  Consider an enterprise build – one that needs to prepare and perform a release.  In the scenario it is essential that the build process contain two separate calls to release:prepare and release:perform. It also might be essential to call out to supporting scripts or other parts of the build that might fall outside of the scope of the Maven build. In a complex build environment, Maven is often not the only build tool involved in a process. Instead of binding our users to a single Maven-specific project type, we implemented Maven support as a builder to make sure that Hudson Professional users had as much flexibility as they needed to create quality enterprise builds.

Creating a New Maven 3 Job If you are new to Hudson Professional and you are wondering how to create a new Maven 3 build, you should follow these simple steps:

1. Create a New Job
2. Select “Build a free-style Software Project”
3. Add a Build Step and select “Invoke” Maven.

Creating a New Maven 3 Project from a Git Repository

The following video demonstrates the process for creating a new Maven 3 project from a Git repository:

Creating a New Maven 3 Project from a Subversion Repository

The following video demonstrates the process for creating a new Maven 3 project from a Subversion repository:

UPDATE :: Enrollment for this class is now closed

Due to popular request, Sonatype has decided to offer a MVN-101 training course in May during European business hours. The training course will be led by Anders Hammar of Devoteam Sweden, a certified Sonatype Training Partner.

Course details:

• MVN-101
• Tuesday May 10, 2011
• European Hours 9:00 am – 4:00 pm CEST (GMT + 02:00)
• ENROLL TODAY!

Maven Mechanics MVN-101 is the premier Java developer training course. It comprehensively covers Maven installation and configuration, explains the motivation behind Maven and gives an overview of related development tools. You will leave this Maven tutorial equipped with a full understanding of the Maven Project Object Model (POM) and a firm grasp of the underlying fundamentals of this development kit including:

• The Maven lifecycle
• Maven plugins and goals
• Multi-module Maven projects
• The contents of the Project Object Model (POM)

Two new virtual training dates have been added to the Sonatype schedule. Learn Maven best-practices from experts in the Maven community. With Sonatype training courses core Maven contributors carefully design our curriculum to cover topics from Maven basics to advanced development infrastructure design.

Registration is now closed for our April classes. Be sure to check out our May offering!

MVN-101: Maven Mechanics

• April 12, 2011
• 11am – 6pm EDT
• Enroll in MVN-101

MVN-201: Development Infrastructure Design

• April 13, 2011
• 11am – 7pm EDT
• Enroll in MVN-201

I recently helped Dennis Lundberg prepare a talk for Jfokus on the top reasons why you should switch to Maven 3. When I work with customers setting up, or improving, their Maven-based development infrastructure I always recommend using Maven 3. I thought I’d share with you the top ten reasons that Dennis highlighted in his talk.

In a nutshell, Maven 3 is superior to Maven 2. While there might not be one single killer feature for most users, it’s more a matter of several bug fixes and improvements that together make it a much better tool. Depending on your use case and your preference, some of these reasons will be more compelling than others. Here’s the list.

#1. Maven 3 is the “Latest and Greatest”

Surely not the wording that you should use when selling this to your manager/boss/product owner/whatever, but when it comes to tools and frameworks many developers like knowing they are using the latest version. Maven 3.0 is where all the new work is done and where the main focus is. Maven 3 has a six week release cycle which was inspired by how things work at Eclipse. This gives you more predictability. You know that critical and serious bugs will be addressed quickly, and you’ll only need to wait a few weeks before the next official release is available.

On the other, in Maven 2.x, you don’t know if there will be a timely release. Bugs are fixed, but there’s no sustained effort to test and push new Maven 2 releases. As an example there are 12 fixed tickets for Maven 2.2.2, of which one has been fixed for more than one year. Few developers are paying attention to Maven 2, and the focus is on Maven 3.

#2. Maven 3 Isolates Project Dependencies and Plugin Dependencies

In Maven 3, project dependencies are retrieved from “repositories” and plugin dependencies are retrieved from “pluginRepositories”. Contrast this to Maven 2 where plugin dependencies could be retrieved from any Maven repository. If you strictly controlled your repositories and locked down your own project’s dependencies you would often run into problems with plugin dependencies. With Maven 3, your own projects dependencies are retrieved from the declared “repositories” while any plugin’s dependencies are retrieved from the declared “pluginRepositories”.

This might not matter to many of you, but I see a lot of larger corporations running into this issue all the time. If you want to have strict control of your own dependencies, you can do this with the Procurement features of Nexus Professional. Procurement is important, but it also requires you to be meticulous about which dependencies you allow from external repositories. With Maven 2, because plugin dependencies can be mixed up with regular project dependencies this forces you to take on the responsibility of enumerating and allowing all of your plugin’s dependencies. In a complex project this mean keeping track of hundred of dependencies which had nothing to do with your own project’s dependencies.

With Maven 3, having these two repositories truly separated makes it possible to define a procured repository and a more open pluginRepository.

#3. Maven 3 Supports Extensions for Alternative Repository Layouts

With Maven 3 it is now possible to create extensions for other types of repositories than the default Maven layout. This gives us new possibilities demonstrated by projects such as Tycho, which allows you to build Eclipse plugins and OSGi bundles with Maven. If you are developing Eclipse plugins, Maven 3 makes it possible to align your OSGi builds with the rest of your Maven-based builds.

#4. Maven 3 is Easier to Embed

One of the main reasons for the major refactoring between Maven 2 and Maven 3, was to allow for Maven to be embedded in other tools. There were some early trials with embedding Maven 2, but it was simply not designed for that and a new architecture was necessary. All of you using m2eclipse are already using Maven 3 and have been doing so for a long time. Switching to Maven 3 for command line executions makes a lot of sense as it aligns the version of Maven in your tools and adds consistency between CLI and your IDE.

#5. Maven 3 has Improved Artifact/Dependency Resolution

Thanks to the refactoring of the artifact resolution logic in Maven, the existing deficiencies of artifact resolution in reactor builds with Maven 2.x have been fixed. I’m sure most of you have run into issues trying to build a multi-module project unless you execute “mvn install”. The reason is that the artifacts built by the modules are not resolved from the reactor but have to exist in the local repository. One very typical symptom of this is having to configure the preparationGoals param of maven-release-plugin to “clean install” instead of the default “clean verify”. Maven 3.0 fixes this and correctly resolves artifacts within the reactor, allowing you to use the default values of the preparationGoals param for example.

More on this topic can be read in Benjamin’s What’s in Maven 3.0 for users? blog post.

#6. Maven 3 has Improved Logging

For Maven 3.0 the logging output has been improved. It’s not a major remake, but smaller changes that makes it easier to read and comprehend. The improvements include separating output from every plugin with an empty line, include the version of the plugin being executed, as well as specifying the artifactId of the project being build. The latter makes the logging output for a multi-module build a lot easier to browse. Should your build fail, the error reporting provides you with a link to a wiki page where the error is described in more details including possible causes.

#7. Maven 3 has More Intelligent Class Loading for Multi-module Builds

Maven uses several different class paths during the build, including separate classpaths for the plugins. Again, this might not seem like a big deal unless you’ve been burned by unanticipated side-effects of what was generally viewed as a bug in Maven 2. In Maven 2 the first execution of a plugin determined the classpath for all executions of that plugin. This often caused issues when the same plugin was used more than once during a build, and the second invocation required the use of a different classpath. In Maven 2, if you needed to execute the Antrun plugin twice in a multi-module build with two different classpaths, you were out of luck. This particular issue was a hassle for the 60 people that voted to fix issue MNG-1323 in the issue tracker.

In Maven 3.0, this issue has been fixed and every execution gets its own, isolated classpath.

#8. Maven has Higher Standards for POMs

Maven 3.0 helps you improve the overall quality of your POM and prevent future problems and errors. Aside from informing of obvious errors and omissions, Maven 3.0 also helps you adhere to best-practices by being more strict about POM validity. For example, Maven 3.0 will warn you if you haven’t specified the version of a plugin used (i.e. using latest release) or have duplicated dependencies with the same POM, as it could result in unpredictable future build results.

Maven 3 doesn’t just have better, more strict POM validation, it will point you in the direction of more concise and complete POM configuration. With Maven 2, it was up to you to gain knowledge of Maven best practices. When you start using Maven 3, Maven will start to provide you with some strong hints about what should be happening in your POMs.

#9. Maven 3 Supports Parallel Builds

Maven 3 didn’t have a lot of new features to speak of as much of the improvement was incremental and behind the scenes. One of the very few new features in Maven 3.0 is experimental support for parallel builds. This support is added to Maven core, but it requires the plugins used during a parallel build to declare themselves thread safe for it to properly work. If a plugin goal is thread safe (and most of the major goals are by now), this is described on the goal description page of its Maven site.

This feature isn’t a match for every build. For example, some types of integration tests might not be suitable for parallel. However, when parallel builds are used, it often results in a much faster build times. Below are two different projects which Dennis did some benchmarking on. The determining factor for performance improvement, aside from the number of CPU cores, was the dependencies between the modules. Project’s with wide internal dependency trees experience more build time reductions than those that had very deep dependency trees. The structure of your internal dependencies dictates which builds must be serialized with one another and which builds can be executed in parallel.

More info regarding parallel builds in Maven 3 can be found on this Maven wiki page.

“mvn package” on Maven SCM trunk (32 modules)

“mvn package” on a corporate project (11 modules)

#10. Maven 3 Builds are Faster, Much Faster

Possibly the one reason all of us could agree on is better performance. Maven 3.0 is both faster and has a lower memory footprint than Maven 2.x. While there might be some rare corner case where this isn’t true, Maven 3 builds are dramatically faster in almost all cases. How much you gain depends on the structure and content of your project though. For his talk, Dennis had some benchmarks which are shown below. The benchmarks were done on a dual-core CPU and 4GB RAM, running Windows XP with JDK 1.5.

“mvn package” on Maven SCM trunk (32 modules)

“mvn package” on a corporate project (11 modules)

Conclusion

If you haven’t already tried Maven 3.0 I strongly recommend you to do so. As a release manager, tech lead, Maven guru, build monkey, or whatever your title is, doing a build of your project with Maven 3.0 will very quickly show you if there are any issues in switching. As installing Maven only requires unpacking a zip file and changing a symbolic link (or environment variable on Windows), it is something you can do in a couple of minutes. Once you’ve experienced Maven 3, I doubt you want to go back to Maven 2.