Brian Fox sat down with Ken Rimple of Chariot Solutions to talk about Nexus and to put repository management in the context of recent developments with Maven. Ken Rimple and Chariot have been long-term partners with Sonatype supporting our Maven training efforts, so Ken has a lot of background about Maven to ask some interesting questions.

Listen to Chariot TechCast, Episode #71, Brian Fox of Sonatype on Nexus 2

The Legacy of Maven: Binary Reusability

The interview leads off with a discussion about Maven, Maven’s history, and some of the recent developments surrounding the Maven ecosystem. Brian identifies binary reusability and declarative builds as the two important legacies of Maven:

“One of the unique things that Maven brought to the table, and what may in fact be the legacy for Maven years down the road, is that it introduced the concept of having binary reusability and not rebuilding the world as everybody was used to doing. The other aspect was making things more of a declarative model especially the dependencies. That was all unique at the time.”

What it takes to Run Central?

Ken and Brian discuss the introduction and development of Central from the beginning of the repository to the current iteration. From the initial efforts to create a single coordinate system for artifacts in 2001 and 2002. Brian’s very involved in the effort to maintain Central so this is your chance to hear Brian discuss some of the internals of the effort: how much bandwidth does Central consume? how much effort is involved in maintaining Central? and what are the day-to-day operations for running Central?

Key quotes about Central: “We’re approaching half a Terabyte for artifact storage” and “We’re seeing 50 to 60 new projects added every day”.

You will hear about how Nexus is used to enforce standards for artifacts added to Central from forges like Apache, JBoss, java.net, Codehaus, as well as the instance of Nexus that Sonatype provides for independent projects: http://oss.sonatype.org.

New Features in Nexus 2.0

Brian then discusses the important features we’re introducing with Nexus 2.0. Including support for .NET, the Repository Health Check, and our support for distributed proxies.

Again, if you haven’t listened to it, you should. Go over to Chariot Solutions and listen to Chariot Techcast Episode #71 – better yet, why don’t you just open up iTunes and subscribe to Chariot’s podcast? here.

You know who this guy is? Probably not, he’s Rube Goldberg.

I’m surprised by how few engineers know his work. Rube Goldberg was a cartoonist who lived from 1883-1970, he’s famous for drawing cartoons of ridiculous and inconceivably complex machines. His work was important during a time in which the world was becoming increasingly mechanized and automated providing a sort of cultural “steam vent” – a way for people to poke fun at machines and industry. I’d embed his work here, but none of it is public domain, so see for yourself or search Google Images. (Be warned, you can spend hours looking at these cartoons.)

I learned about Rube Goldberg from an Engineering professor who, at the time, said, “Rube Goldberg is the most important thing you’ll learn over the next four years”. Back then, we all thought he was joking, but it turns out that he wasn’t. In fact, I wish more people, especially “build engineers” had some exposure to these cartoons. If they had, they’d take a step back and realize that there has to be a better way.

“accomplishing by complex means what seemingly could be done simply”

The title of this section is the Webster’s New Dictionary’s definition of “Rube Goldberg”. Does this describe your internal software distribution process? I fear it might. I’ve seen some builds and deployment procedures that convince me that people are actively trying to create new Rube Goldberg machines with SCM tools and build scripts. Almost every month I see some new variation of what I eventually characterize as a “Crazy Rube Goldberg” approach to releasing software. Systems that barely work, and that no one dares to change for fear of breaking some crazy contraption.

Goldberg’s cartoons and the build’s inspired by them share common characteristics. The complexity of his cartoons are almost always defined by the complex ways in which some object is transported from one “stage” of the cartoon to the next and the comic “indirection” involved in every process.

A cartoon example, a man slips and falls, stepping on a switch which then activates a mechanical hand that throws a bag of sand on to a scale which then triggers a mechanism to fire an arrow at a balloon. A build example, Jeff runs a build on his laptop because that’s the only place release builds work properly, he then checks the WAR into Subversion and then sends a series of emails to several other people. A deployment engineer then takes the output of Jeff’s build and runs it through a custom, somewhat confusing Perl script to customize a series of configuration variables.

Common Characteristics

The common characteristics between Rube Goldberg cartoons and Rube Goldberg builds are:

• Unnecessary Indirection

  • Cartoon: Instead of just creating a machine to pop a balloon, a cartoon creates a contrived process that involves unnecessary (and often impractical) indirection involving pulleys. Goldberg’s cartons often find inventive ways to convert potential energy to kinetic energy in some fantastic way you wouldn’t have thought of.
  • Awful Build: Instead of using a technology to keep track of binary artifacts, a Rube Goldberg build overloads SCM or (worse) email to send multimegabyte binaries around. Instead of using a technology with built in support for automatic email notification, your build relies on some ad-hoc system of people remember to send just the right individuals a set of process-related emails.

• Reliance on the Unlikely

  • Cartoon: A bag of sand thrown across a room lands exactly on a bulls eye of a target triggering yet another unlikely series of events, continue, repeat…
  • Awful Build: A complex build that takes more than an hour to complete has a random dependency on a web service that provides an series of configuration variables used to configure a binary for production. If the web service happens to be unavailable, or if it returns invalid results, the entire build process must be started from scratch.

• Reliance on Animals (or Humans)

  • Cartoon: One of the comic elements almost always present in a Goldberg cartoon is the presence of an animal of some sort. A bird who, when hit with an arrow squawks loud enough to scare a dog into running and subsequently lifting a weight via a pulley system.
  • Awful Build: Awful build systems frequently require direct human intervention to the point where the build doesn’t work unless the same individual is always “strapped” into the build in just the right way. “After that compilation step completes, Jack usually logs in and modifies the build number before he runs the deployment script”.

• Comedy

  • Cartoons: Rube Goldberg’s cartoons are comedic because they involve the impractical or the impossible. Readers find them amusing because they are plausible, yet inadvisable ways to implement a process.
  • Rube Goldberg-style builds are comedic because they involve the impractical or the impossible. Developers find them amusing because they are plausible, yet inadvisable ways to implement a process.

Build/Deploy systems are Comedy?

If they work, they can be. It’s quite common for the build/deploy lifecycle at a corporation to involve a series of steps that no one fully comprehends. Maybe there’s a crazy 4000 line Ant script or maybe there’s a multi-headed Hydra of bash magic that happens to produce a working system. In the last few months I’ve talked to people with build systems that work, but no one on staff understands how. When you ask about these release procedures people often chuckle a bit – “Oh, right, our release process…. well let’s say it’s a little confusing…”.

“Why does the build do it like this?” is often met by, “I don’t know, but I’m too scared to change this thing right now, the contractor left us with this and it works… for now.” In other words, many of these builds involve a series of unknown pulleys and unexpected indirections that resemble a cartoon.

In the next part of this series, we’re going to talk about how Nexus can be used to untangle a build into something more reasonable. Something that doesn’t involve pulleys, magic, and a series of unlikely events.

Note: Rube Goldberg doesn’t do it for you? Then just watch the OK Go video “This Too Shall Pass” to get a modern idea of what I’m talking about. If you haven’t seen that, you should. In fact, if you haven’t seen this video yet, stop reading our boring Nexus blog and take a break, it’s worth it.

Some questions about syncing Maven repositories between two sites were recently asked on GetSatisfaction.com.

“We will be moving data centers and want to setup another Maven2 repo that is managed with Nexus OSS. We want both repositories to be online and read\writeable until we migrate all our environments to the new site. I have a few questions:

1. What is the best method for copying the repo to the new location?
2. What is the best method for keeping the two repos in sync? We want to minimize network bandwidth usage.”

The video below answers these questions, and offers multiple solutions:

Next week, Jason will be presenting the webinar: Improving Your Java Development with Apache Maven 3 and Hudson. All registrants will receive access to the recording after the event so if something comes up and you can’t make it, you won’t be missing out.  Be sure to register today!

Improving Your Java development with Maven 3 and Hudson

Tuesday, June 7, 2011 @ 10:30AM – 11:00AM EDT (UTC/GMT – 0400)

Presenter: Jason van Zyl, Sonatype CTO & Founder

Attend this webinar to learn about the advantages of upgrading to Apache Maven 3, including improved speed, greater stability and increased compatibility. Jason will also talk about the greatly improved support for Maven 3 within Hudson that is easy to configure and supports complex build scenarios with ease. We will cover the Eclipse IDE integration for both Maven and Hudson that improves developer productivity.

Register now!

As part of the Eclipse Live series of webinars, Sonatype software developer Pascal Rapicault is giving a presentation on Tycho: Building Eclipse plugins with Maven.

About the webinar:

Tycho is a set of Maven plugins and extensions for building Eclipse plugins and OSGI bundles with Maven. Eclipse plugins and OSGI bundles have their own metadata for expressing dependencies, source folder locations, etc. that are normally found in a Maven POM. Tycho uses native metadata for Eclipse plugins and OSGi bundles and uses the POM to configure and drive the build. Tycho supports bundles, fragments, features, update site projects and RCP applications. Tycho also knows how to run JUnit test plugins using OSGi runtime and there is also support for sharing build results using Maven artifact repositories.

Join this webinar to get an overview of the Tycho project and to learn what plans the project has for the future.

• Date: May 3, 2011
• Time: 9:00 am PST / 12:00 pm EST / 4:00 pm UTC / 6:00 pm CET
• Length: 60 minutes
• Enroll here!

As mentioned earlier on the Sonatype blog, we’re taking some of our most popular sessions from EclipseCon 2011, and releasing them to the wider developer community. The second installment from EclipseCon 2011 is m2eclipse: The collaboration of the Maven & Eclipse Platforms.

Software developer Igor Fedorenko details the new features and changes to m2eclipse 1.0, including pom.xml editor enhancements and reworked build lifecycle mapping.

For more videos from the Sonatype team, visit our Resource Center, or go to our YouTube channel.

If you missed Sonatype’s latest webinar, the Future of Hudson Continuous Integration, you can catch the replay right on Sonatype.com.

To view the replay, click here.

Our next webinar – Hudson Continuous Integration with Sonatype Professional – is taking place on March 29. If your development team is looking for best practices for integrating Maven, Hudson and Nexus Professional this webinar is for you.

Sonatype Professional is the only integrated suite created for Maven-centric builds. The suite includes Sonatype Certified & Integrated Distributions of: Maven, Hudson, Nexus Professional, Eclipse Integration, Developer Onboarding as well as support from the experts. Register to learn how Sonatype Professional empowers development teams to realize the promise of agile through continuous integration, while reducing project risk.

• Date: Tuesday, March 29, 2011
• Time: 9:00 am Eastern Daylight Time (New York, GMT-04:00)
• Duration: 30 mins
• Presenter: Blaine Mincey, Sonatype Senior Systems Engineer
• Register for this webinar

Another year of great talks and Eclipse workshops are just around the corner!

EclipseCon 2011 is being held in Santa Clara, California from March 21 – 24.

As a Strategic Member of the Eclipse Foundation and as a proud Gold Sponsor, Sonatype is looking forward to another year of great talks, tutorials and BOF’s at EclipseCon 2011. Sonatype technical leaders will host a number of sponsored talks on Tuesday, March 22 in the Cypress Room. Topics we will cover include: Next Generation Development Infrastructure, Hudson Continuous Integration, Tycho Build Conversion, Discovering p2 API’s, Maven 3 and many more.

Check out the full lineup of presentations here.

We hope to see you there! But if you can’t make it, get all of the conference updates from the Sonatype team on Twitter by following the hashtag #EclipseCon.

Event Details:

• Date: March 21-24, 2011
• Location: Hyatt Regency Santa Clara, CA
• Event Website: http://www.eclipsecon.org/2011/

Thank you to everyone who attended Sonatype’s recent webinar, Improving Developer Productivity with Sonatype Professional. If you were unable to attend the webinar recording is now available.

In a recent survey, 80% of developers, managers and architects said lack of integration between their tools was a huge efficiency drain. As teams standardize on tools like Maven, Hudson, and Nexus integration headaches can be real productivity killers.

View this webinar to learn how Sonatype Professional can help your team spend less time chasing problems and more time delivering code.

In Maven 3.0.3, we introduced some improvements to the way plugin goals can be configured. This post summarizes these improvements and gives some examples of how Maven plugin configuration has been enhanced. We’ve focused on making improvements that will reduce the friction of plugin configuration for both plugin developers and Maven end-users.

First, a general tip for users interested in using these enhancements in POMs: use the requireMavenVersion rule from the Maven Enforcer Plugin and required Maven version 3.0.3.  This will save other team members from running into strange build failures due to plugin misconfiguration if they are still using previous Maven versions.   Likewise, plugin authors that take advantage of these enhancements should properly declare the Maven prerequisite in the plugin POM.

Generic collections

The plugin configurator can now employ the generic type argument of a collection parameter to determine the type of the elements. In practice this means plugins can use collections as input parameters more freely without having to worry about the user being forced to specify the proper element type in the POM.

For instance, the goal parameter:

/** @parameter */
java.util.List<java.io.File> files;

can be configured like this in the new Maven version:

<files>
  <!-- Maven 3.0.3 can take care of converting this to a java.io.File instance -->

  <file>pom.xml</file>

  <!-- The previously required implementation attribute is now superfluous but still works -->
  <file implementation="java.io.File">build.xml</file>
</files>

Instead of having to specify the type of each element, Maven 3.0.3 will now automatically convert the first element in the previous example to an object of type “java.io.File”.

In absence of the implementation attribute, Maven usually looks for a class named <goalPackage>.<xmlElement> to determine the element type. The recognition of generics now enables plugin authors to use bean classes for collection elements that reside in a different package than the plugin goals.   In summary, collections now provide the same ease of configuration as classical arrays.

Automatic conversion of arrays to collections, and vice versa

Continuing in the spirit of the previous improvement, the new plugin configurator will now automatically convert a collection obtained from a parameter expression to an array if that’s the type expected by the plugin parameter. Likewise, an array would be converted to a collection if needed. The bottom line is that plugin authors targeting recent Maven versions can freely decide to use an array or a collection for a plugin parameter regardless of whether its default value actually yields a collection or array:

/** @parameter default-value="${project.compileSourceRoots}" */
List<String> sourceRootsAsCollection;

/** @parameter default-value="${project.compileSourceRoots}" */
String[] sourceRootsAsArray;

Before this change was made to the configurator plugin developers and Maven users had to be very careful about passing an array to a method that expected a List or vice versa. While this might seem like a minor change, it is one of many that will make plugin development easier.

Configuration of collection/array via system property

For many plugin parameters it is occasionally convenient to specify their values from the command line via system properties. In the past, this was limited to parameters of simple types like String or Boolean. The latest Maven release finally allows plugin users to configure collections or arrays from the command line via comma-separated strings. Take for example a plugin parameter like this:

/** @parameter expression="${includes}" */
String[] includes;

This can be configured from the command line as follows:

mvn <goal> -Dincludes=*Foo,Bar*

Plugin authors that wish to enable CLI-based configuration of arrays/collections just need to add the expression tag to their parameter annotation. Note that if compatibility with older Maven versions is to be kept, the parameter type must not be an interface but a concrete collection class or an array to avoid another shortcoming in the old configurator.

Inlined collections

Many users complain that Maven POMs make heavy use of container elements for collections. Instead of just listing a series of dependency elements, you have to wrap all of your dependency elements in a dependencies element. The same is true for includes and excludes and other elements throughout the POM.  This extra redundancy often adds up to some very large POMs. In Maven 3.0.3, we’ve made a change to the plugin API that will support plugin configuration without requiring container elements for collections.  Here is an example:

<fileset>
  <directory>src/demo</directory>
  <include>Foo*</include>
  <include>Bar*</include>
  <exclude>*Bad</exclude>
</fileset>

Previously, Maven would only look for a field or setter when processing the <include> elements. Now it also looks for an adder. So to support the above configuration, a plugin author would need to implement the configured bean like this:

public class Fileset
{

    private List<String> includes = new ArrayList<String>();

    public void addInclude( String include )
    {
        includes.add( include );
    }

    ...
}

The adder needs to be named add<xmlElement>() and must be public, non-static and have a single argument; it may have a return value. In case where multiple such adders with the same name exists, it is undefined which one gets actually called so be sure to avoid overloading the adder.

Given that merging of plugin configuration during POM inheritance or profile injection is purely based on the XML structure and not guided by information about the actual data types, users that employ this compacted configuration are well-advised to not mix it with the usual collection-style configuration format for a given plugin parameter as the merged configuration likely doesn’t produce the intended result.

Bean default properties

We can take the idea of inlining configuration a little further. Have a look at this example configuration:

<resources>
  <resource>
    <directory>src/foo</directory>
    <filtering>true</filtering>
  </resource>
  <resource>src/bar</resource>
</resources>

Looks odd at first, doesn’t it? Obviously, the <resource> element describes some complex structure, but the second <resource> element in the example consists just of a mere string. It’s not hard to guess that<resource>src/bar</resource> could be a shorthand form of <resource><directory>src/bar</directory></resource> by assuming the <directory> element is the default/primary property of a <resource>. With the new plugin configurator, this is actually possible now.

To enable the feature, a plugin author would have to add a method called set() to the bean class in question:

public class Resource
{

    private File directory;

    public void set( File directory )
    {
        this.directory = directory;
    }

...
}

This set() method must be public, non-static and take a single argument; it may have a return value.

Besides saving users a few bits of XML, this feature can also be used to configure complex beans via system properties from the command line. Let’s say we have a plugin parameter like this:

/** @parameter expression="${artifact}" */
Artifact artifact;

Now further assuming the Artifact bean used here implements a nice set() method that is smart enough to parse a string into artifact coordinates, one could invoke the plugin directly via:

mvn <goal> -Dartifact=org.apache.maven:maven-core:3.0

Last but not least, the default property feature also enables a smooth upgrade path in case plugin parameters need to be changed from simple values to complex structures. Going back to the initial example with the <resource>elements, consider the plugin author originally didn’t anticipate the <filtering> element and designed the plugin parameter to be of type File[].

With previous Maven versions, the only way to extend the plugin to support the<filtering> element is to deprecate the existing <resources> parameter and introduce a new parameter using the complex type or alternatively just break compatibility with existing POMs that use the old plugin version. Using the default property support, a plugin can change the parameter type from File[] to Resource[] without affecting existing users.

Simpler configuration of properties parameter

While java.util.Properties is just a concrete implementation of a Map, the configuration required for parameters of type Properties is structurally different from the configuration format for Map parameters. For the sake of consistency and conciseness, plugin parameters of type Properties can now also be configured like a Map, that is:

<properties>
  <key1>value1</key1>
  <key2>value2</key2>
</properties>

User-specified implementation class for map parameters

Previous Maven versions always used a TreeMap to configure parameters of type Map. In other words, neither the user nor the plugin author had any control over the map implementation being used. Just like with the collections, the updated plugin configurator recognizes an optional implementation attribute:

<map implementation="java.util.LinkedHashMap">
  <key>value</key>
</map>

In absence of both the attribute and a concrete implementation class in the parameter declaration, TreeMap continues to get used.

Hexadecimal and octal numbers

In some contexts, it’s more convenient to specify a number in hexadecimal or octal notation. To support this, plugin authors previously had to declare the corresponding plugin parameters as strings and do the conversion themselves. Now, it’s natively supported, using the prefix “0x” to denote hex notation and the prefix “0″ to denote octal notation.

<rgb>0xFF00C0</rgb> <!-- hex number -->
<perms>0664</perms> <!-- octal number -->