Due to high demand, we have added a second webinar presentation next Wednesday at 2PM EDT (GMT-0400) to accommodate multiple time zones. Here are the details for the presentation:

Join Brian Fox this Wednesday, May 23 at 11AM EDT or 2PM EDT (GMT-0400) for a 30 minute tour of Insight for CI. In this demo, Brian will show how Insight for CI will help you:

• Generate a detailed bill of materials for every build in Hudson and Jenkins.
• Find and fix license, security and quality problems quickly.
• Set rules to notify you of problems, fail builds, or establish workflows.

If you register, you’ll also receive access to the recording after the event. So if something comes up and you can’t make it, you won’t miss out.

Two sessions are now available on Wednesday, May 23. Choose the best time for you:

Register – 11:00AM EDT (GMT-0400)

Register – 2:00PM EDT (GMT-0400)

One of my responsibilities at Sonatype is creating the pages that communicate licensing and security information in Nexus Professional and Insight for CI. We have a large team that is responsible for these pages and making sure that we’re providing accurate information. You would be surprised at the number of interesting edge cases that we identify in the process of scanning 400,000+ artifacts in Central. From invalid licenses to exotic, one-off licenses that include odd requirements, everyone who works on this team has had to become an expert in OSS licensing.

The following post about Meteor, a new, node.js-based approach to building web applications backed by MongoDB got my attention because it highlights some of the tricky integration issues we’ve had to think about when coming up with hypothetical use-cases for Insight. Here’s a quote that captures the complex relationships between Meteor, originally a GPL-licensed Javascript library, and an Apache-licensed library to access MongoDB:

From Olov Lassus’s popular blog post “Meteor meets NoGPL”

“The copyleft (viral, contaminating, whatever you want to call it) aspect of GPL is tricky. Take MongoDB as an example. Meteor uses it by importing the node-mongodb-native package (require(‘mongodb’)). That one is Apache 2.0 licensed, which is a permissive license that happens not to be compatible with Meteor’s GPL (v2) license, at least not according to the FSF. Tricky. Dependency chains, bindings between JS ↔ C and RPC makes it trickier even. I wouldn’t be surprised to see Meteor change to a GPL + a-bunch-of-OSS-exceptions license similar to what Qt and MySQL used to have, to avoid issues like this.”

Don’t get me wrong, I’m not questioning the Meteor team’s right to choose whatever license they want, but I noticed this post because these are the kinds of relationships that we’ve been trying to sort out between different libraries in Central. We’ve encounter libraries that advertise themselves as BSD-style licenses which end up requiring dependencies on GPL components. This highlights the problem of licensing intent versus licensing reality. Just because a particular components is licensed under a particular license doesn’t mean you can actually use it under the terms of that license.

Lots of activity on Olov Lassus’ twitter feed and many opinions on the hacker news threads.  This one was good about the ambiguity of the GPL w.r.t. derivative works.

Note: Meteor has since changed the license to MIT, which makes this very interesting project that much more compelling to a wider audience.

As we’ve been busy building out the Insight product line we’ve spent significant time considering the issues associated with “conflicting” and “invalid” licenses — licenses which upon consumption preclude further redistribution without being in violation of the licensing terms.  Conflicting (or incompatible) licenses are problematic for development organizations using open source software as there is no effective way to consume and then redistribute the software (or derivative work).  You simply cannot combine GPL and EPL 1.0, for example, because it is not possible to maintain compliance with all licensing obligations specified by both under any licensing construct upon further distribution.  EPL cannot be consumed within GPL and vice versa.  See http://www.gnu.org/licenses/license-list.html#EPL for additional information.

If you consume both EPL and GPL in a Maven POM or another build, and then you subsequently ship that software, you would not be able to satisfy your obligations as a distributor and would therefore be in violation of one or both of the licenses.   As developers, we have enough to worry about already.  This a job best done by the tools we use — in this case Insight for CI and Nexus Professional.  Depending on your circumstances, having your CI system alert upon detecting incompatible licensing constructs at build time reduces risk and costs by catching problems early in the development lifecycle.

Invalid license constructs are similar to conflicting licenses in that they also create a situation where a distributor is not able to meet all of the required license obligations specified by all open source components included.  The difference is that the licenses are in fact compatible, just that they are not valid in the particular direction specified.  For example, you cannot move from nonpermissive to permissive, such as GPL to BSD.  But, using BSD licensed code within a broader GPL work is perfectly acceptable.

An example scenario:

Someone integrates a GPL licensed artifact into their code but (incorrectly) chooses a BSD license for the combined work.  This is an invalid license structure since the GPL is a non-permissive license and BSD is.  The GPL does not allow for a reduction in the rights expressly granted to the consumer, which is what is implied by a BSD license — e.g. as a consumer of BSD, you are not entitled to receive a copy of the source code from the distributor but with GPL you are.  There is no way, under the BSD license, for a distributor to fully satisfy the obligations of the GPL, hence this arrangement is invalid.

As part of developing this product, we’ve consulted with legal professionals who have given us guidance on handling invalid and conflicting licensing.  The short version:

• Sonatype’s licensing products only report factual information.   Our products don’t give specific legal advice.  (In other words, Insight for CI and Nexus Professional are not a substitute for a qualified legal professional.)
• Obligations shown for an artifact must be the set of all obligations of everything in a particular JAR regardless of hierarchy.   This is why you might be surprised every once in awhile by an artifact that advertises itself under an Apache 2.0 license that is listed as a potential LGPL or GPL artifact.    Licensing isn’t about intent, it is about what the end product’s resulting obligations are given a combination of artifacts covered by various licenses.
• Sonatype products effectively identify “invalid” and “conflicting” licenses and highlight these occurrences.   If there is an artifact in your repository that is covered by two licenses that cannot be combined that’s something we will flag so that you understand the potential distribution risks implied.

Such a situation has an element of risk for a customer.  We identify such risks, and call attention to the potentially problematic combinations.   Again, our products are not in the business of rendering legal opinion, we’re just here to report the facts.  It is an exercise for the user to determine the implications this might have given their unique circumstances.

Most developers don’t spend much time thinking about these types of licensing issues.  Who needs more work to do?  Yet, every day, developers consume open source components often without giving these considerations much thought.  This means that when an organization ultimately decides to take a look at licensing they can uncover issues they never knew existed, occasionally nestled deep within their application’s dependency graph.  Often times, there is also no path to remediation since the work in question has already been distributed.   In short, you could be compelled to meet the obligations of hidden viral licensing — perhaps through open sourcing your intellectual property.

I was doing a bit of data analysis of the data that drives our Nexus Professional popularity results and I came across some statistics that show demand for Google Guava has been picking up over the last year. Our Top 10 list for general utilities contains the usual suspects. Libraries like Commons Lang and Commons Beanutils are predictably near the top of the list as are both log4j and slf4j. Not only are these the utilities you’d expect to see in almost every Java project, many of the dependencies you depend on also reference these libraries. This list is a list of utilities and projects you’d better be familiar with if you are programming in Java because you will undoubtedly encounter them.

Here is a list of the Top 10 Utilities from April 2012. Note how Google Guava jumped three places from #15 to #12 with a 2.5% increase in demand from March. While I don’t expect Google Guava to surpass the popularity of Apache Commons components any time soon, it will be interesting to see if Guava becomes a standard that challenges Commons Lang. Guava, like Apache Commons, is a collection of utilities and classes that supplement Java, while they have overlapping purposes, I tend to continue to have both on my classpath whenever I’m coding.

Caveat: I’m comparing utility libraries with the exception of JUnit. JUnit is downloaded automatically by a number of tools (tools that don’t appear to cache artifacts between instantiation). Because of this JUnit downloads are off the chart. If you average out the data, JUnit is being downloaded approximately once a second (across the entire month).

A big thanks to all of you who registered and attended our sneak preview of Insight for CI last week. We had a great turnout and a lot of fantastic questions! If you didn’t have a chance to register, that doesn’t mean you have to miss out. The replay is now available.

Request the webinar recording here.

Thank you!

There are over 400,000 components in the Central repository including everything from servlet containers like Apache Tomcat to critical application infrastructure like Spring and Hibernate.    When you are designing an application or trying to update an application’s dependencies, how do you choose which component to use?

Here’s an example of a decision you may have to make in the next few months.    Assume you have the chance to use a newer version of Spring, evaluate Hibernate vs. iBatis, and adopt a new REST-friendly web framework.   For each of these new and updated components you are going to have to ask yourself three questions:

• Which version of the library has the largest “install base”?  It often doesn’t make sense to use the latest version of a component, especially if it is a major release.   If you are looking to reduce risk, don’t code on the “bleeding edge” of technology.  Use the most popular version of a component.
• Which version of the library is free of security vulnerabilities?  The only thing worse than getting hacked is realizing that you got hacked because you weren’t paying attention to known vulnerabilities.   If you are upgrading to a new version of a library, make sure it is secure.
• Which version of the library is compatible with your OSS license policy?

Nexus Professional 2.0.4 brings the answer to all three of these questions to the search results.  Here are the search results showing the results for tomcat-catalina.    We’ve combined popularity data from Central with security and licensing information.

Without the popularity data you might have just selected the latest version of the library, version 7.0.27 which has been available for 37d.   If I were selecting components for an application, I would likely stick with Tomcat 7.0.25 based on the relative popularity of the artifact alone.   7.0.25 is, far and away, the most popular artifact of this group.

Sonatype’s Nexus Professional is the only product that incorporates popularity data directly from Central.   If you are interested in using Nexus Professional to evaluate your dependencies, download a copy and start your trial today.

As everyone probably knows by now, we recently announced the findings of our annual survey of developers, architects, and managers.   We were fortunate to have more than 2,500 responses to this year’s survey.   If you missed it, you can find the details in the press release.

Many of you have requested access to the individual graphics we produced for the findings so you can link to and share individual slides.   Here are the survey results:

Click here to view the Survey Results

We put together a slick little gallery where you can access and share any of the individual images.

Get a cool mug and a chance for a day with Jason

*All entries must be received by April 29, 2012

Tell us how Nexus has helped your development organization and you’ll be entered into a drawing for one day of onsite build and repository consulting with Jason van Zyl, Sonatype CTO and founder of the Apache Maven project. And everyone who enters will also get one of these wicked awesome Sonatype Nexus travel mugs.

It’s easy. Just copy and paste the questions below into an email and send your answers to sonatypestories@sonatype.com. We’ll publish your stories on our blog.

We’ll need your name, role, company and shipping address along with answers to the following questions:

1. Are you using Nexus OSS or Pro?
2. What was life like before Nexus?
3. How is your team using Nexus?
4. What value did you see after using Nexus?
5. What would you tell somebody considering using Nexus?
6. Is there anything else you would like to tell us?

Contest rules can be found here.

Here’s a message to nexus-user from Eric Kolotyluk an active Nexus user who just upgraded his instance of Nexus and sent this message to the Nexus Users mailing list:

OK, I had to move my Nexus to a different server.

1. I installed the latest version of Nexus Pro on the new server.
2. I copied over sonatype-work from the old server to the new server
3. I installed the license
4. It just works.

Great work Sonatype for continuing to improve the ease of administration.

Cheers, Eric

Sonatype invests an huge amount of effort in unit testing, integration test, and regression testing. Engineers are never pressured to sacrifice quality to meet a deadline, and our Engineering team goes out of their way to make sure that upgrades and installations are as straightforward as possible. It’s great to get feedback like this, especially when it contains one of the more famous Apple, Inc slogans: “It just works”.

While this user had a seamless upgrade experience, we still recommend that everyone installing and upgrading Nexus read both the Nexus 2.0 Release Notes and the Nexus Upgrade Instructions.

We’re often asked by customers to prove that Nexus can scale to meet the demands of thousands, and sometimes tens of thousands, of developers. Fortunately, we don’t have to stand up an expensive set of machines for a proof-of-concept as we have the world’s largest collection of active open source projects hosted on a single instance of Nexus Professional running at http://oss.sonatype.org. This instance isn’t just proof that Nexus Professional can scale, it serves as a public instance that you can model your own instance after.

If you are looking for an estimate of the hardware required to support your instance of Nexus, this post will detail the configuration and specifications of the Nexus OSS repository instance. This instance is the largest known deployment of a repository manager in active use.

Performance of Nexus OSSRH

Nexus OSSRH serves requests on the order of 1,400-2,500 requests per minute. What drives this level of activity? First, the instance serves as a snapshot repository for many open source projects. If you look at the list of projects hosted on OSSRH, it is a large list. As we examine the logs for oss.sonatype.org we regularly see thousands of unique IP address every day, and oss.sonatype.org is involved in a number of OSS project’s CI builds. This means that at any given time, OSSRH is supporting any number of simultaneous CI builds and over the course of a given day we’re serving artifacts to thousands of developers.

OSSRH approximates the performance characteristics required for the largest development efforts in the world: with multiple geographic locations, 24/7 uptime requirements, and very high performance standards. This service has to stay up. If OSSRH were to become unavailable, you would hear an immediate outcry from every affected OSS developer. Just choose a day and search for projects announcing that they’ve pushed artifacts to oss.sonatype.org on Twitter and you’ll see that every day has several critical releases.

When a customer asks us to prove that Nexus Professional scales, we don’t have to stop and setup a contrived performance test. We support this level of activity every single day. All we need to do is point them at OSSRH.

Nexus OSSRH Specifications

We’ve established that OSSRH is at the center of a large amount of active OSS development. It serves between 1400 and 2500 requests per minute, and it is a mission critical resource. It would be reasonable to expect that this service runs on a cluster of machines distributed throughout the world to minimize latency. Think again, this is a single VM with modest specifications running at Contegix and constantly monitored by New Relic.

Our standard setup for all managed forges is:

• 2 CPUs
• 3GB RAM
• 400GB disk (this is completely dependent on your repository contents)
• RHEL 5.6 x64 (Contegix, our managed hosting service, recommends using this OS)
• Java 1.6 x64 with 1GB Heap* (see correction below)
• The virtual disk is located on a SAN connected with iSCSI over 1GBE

If you are supporting a global-scale network of thousands of developers, the hardware cost for this Nexus instance is a “drop in the bucket”. The specifications for one instance of Nexus Professional running on a service like Amazon EC2 would easily fit on an m1.large instance with space to grow or a very modest VM. (The only thing you might spend on is the disk requirement. For OSSRH, we have a six-disk RAID 50 approach described below.)

Scaling Nexus: I/O Requirements, Network, and Disk

Under heavy load, increasing the number of CPUs and amount of RAM may help, but often the gating factor is either disk I/O or network. We do not recommend using NFS to mount a virtual disk for the working folder as many customers have had trouble with locking and corrupted indexes. iSCSI is working very well for us on oss.sonatype.org and it also works for many of our flagship customers.

Over the course of a day, the system typically needs to scale up in terms of network and IO. And, Nexus “sings” under heavy load because we have made numerous code-level optimizations to ensure that we’re making effective use of caching to reduce roundtrips to disk. For I/O performance, we recommend a redundant solution that maximizes disk spindles, while maintaining fault tolerance. We use RAID 50 in our SAN. A RAID 50 combines the straight block-level striping of RAID 0 with the distributed parity of RAID 5. It is a RAID 0 array striped across RAID 5 elements. This approach emphasizes both performance and extreme reliability, it requires at least 6 drives.

If you need scale, Try Nexus Pro

Sonatype designed Nexus to meet the demands of the OSS community from the beginning. We’ve been supporting global-scale OSS communities for years, and we’ve integrated the lessons learned from supporting active OSS development into Nexus Professional. If you need to scale, try Nexus Professional Today.

Correction from Mike Hansen: With 2.0 we upped that to 2GB, at least on OSSRH.  But that pretty much just provides some extra headroom…  Actually, IIRC, the reason we went to 2GB was because we were battling memory consumption with some repository indexes that had not been optimized (i.e. the index optimization task had not been run for a very long time).