My Assumption: Most of us don’t read documentation…

Based on my experience, I’m going to guess that most of us likely just dive into implementation when it comes to new tools and libraries. Even though one of my focuses is documentation, I’m only assuming that 20% of you read anything in the book prior to install. Sonatype understands this and with every Nexus release our engineers and product managers are trying to implement new features to be as straightforward as possible.

This isn’t to say that we’re no longer writing documentation. Documentation is essential when you are setting up critical, enterprise-grade development infrastructure. In my time consulting and training I’ve seen some really “interesting” builds that have developed from what I’m calling the “Just Dive In” approach: what happens when an organization adopts a tool before it understands what the tool provides or even what it is. What happens when you never take time to learn about a tool.

The “Just Dive In” Approach

This is a common malady for programmers, and we’re almost all guilty of this approach at some point in our career. Someone tells you to check out Hudson or Maven, you read some of introductory articles on DZone, and you immediately dive in. Even though you might only have a rudimentary grasp of the tool, you’ve converted your build over to Maven and Nexus. It’s full of duct tape because you didn’t fully grasp the Maven lifecycle, or you don’t use snapshots because you didn’t have time to learn about the differences between release and snapshot artifacts.

Note: I’ve been there. This is how busy developers adopt new technologies. You adopt a new technology like Spring or a new web framework by doing. Having time to read a full length book on a new technology is rare. My own theory is that it always takes two attempts to adopt a new technology with the first attempt being a dress rehearsal full of mistakes and missteps. So back to our example…

Maybe you make a mental note to purchase the book or take a training course on the technology… later, when the project calms down, but time moves quickly these days. Deadlines and meetings multiply and suddenly you find yourself unable to take some time out to really learn Maven or Nexus. All of a sudden your half-baked prototype become a production dependency.

Hoping you can swim…

As a consultant this is usually when someone calls me up and needs some emergency help. They migrated to Maven (or they set up Nexus) but they didn’t take a day to start from the fundamentals. A product deadline is approaching and someone needs to do the equivalent of open heart surgery on a non-standard build in the matter of days. Often the changes that need to be made to fix a build full of bad assumptions and introduce more compromises and workarounds. This is expensive, last-minute work that could have been avoided with training.

While most of us don’t read documentation, I think it is up to managers to make sure that someone in the organization has a grasp of the fundamentals from the very beginning. That’s my cue to tell you about our Nexus training class, it’s a comprehensive introduction to Nexus best practices and it walks you through the decisions you’ll need to make *before* you introduce a repository manager into your architecture. (Let’s see that’s a couple hundreds dollars vs. the price of an emergency consulting engagement – if you have a complex environment, I’d recommend the training first.)

Conclusion: Check out that pool before you dive in…

This rapid adoption trend has become more apparent in the last decade. As an industry we’re trying to create developer-facing APIs, tools, and frameworks that are simpler to adopt and use. While you should be able to get up and running with an evaluation of Nexus or Maven in a few minutes, we can’t let this simplicity keep us from a fundamental truth. When you don’t set aside enough time to learn a new build tool or piece of development infrastructure you end up with these half-migrated disasters that end up having to be rewritten often at the last minute.

If you want to avoid this situation, set aside some time to understand products before you dive into a production implementation. Understand the fundamentals of repository management and dependencies before you implement an enterprise-class build and Nexus deployment. Resist this urge to just get started and read the introductions to some of these books we’ve invested in before you start implementing a deployment. It’ll save you time and money, I promise.

Join us Tuesday, February 21 at 11:00AM (GMT-0500) for a 30 minute demonstration of Nexus 2.0, with Jason van Zyl, Sonatype Founder & CTO. You’ll see all the new features and learn how Nexus 2.0 will help you:

• Avoid downtime by using a highly available architecture
• Improve repository management with enhanced component information
• Standardize on a single repository for .NET, Java, and OSGi

Reserve Your Seat

If you register, you’ll also receive access to the recording after the event. So if something comes up and you can’t make it, you won’t miss out.

“Wait, that’s how Central works?”

There’s something so automatic about dependency management in Maven that it often takes people a few months to understand exactly where those JAR files are coming from. In an 8 hour Maven class, I get to dependencies in the third hour, and after describing Central, what it is was like before Central, how metadata is stored in a repository alongside binaries, transitive dependencies, etc…. it all falls into place, and people realize that this simple thing they’ve grown accustomed to is only easy because of a ten year effort to refine the model, the creation of a support structure for source forges at places like Oracle and Google, and a constant investment in infrastructure.

On the one hand, it’s a great success that Central is, for the most part, an invisible utility that supports developers.  On the other hand, it’s the kind of thing that people can start to take for granted very easily.

For example, a few months ago I spoke to someone who worked in an environment disconnected from the internet for security reasons.   This individual was talking about how limiting it was to have to download JARs from open source projects manually and assemble them in a project.   His words were: “It’s like programming Java in 2001 again.”

How can you help?

Imagine millions of developer spread all over the world: different time zones, different applications, but they all hit the same service: Central.    Some regions have more developers than others so we certainly see peaks in usage throughout the day, but in general, Central’s serving thousands of files throughout the world at any given time during the day.

Maybe someone just installed Maven for the first time, or maybe they blew away a local repository, with numbers like these we see a world that has a constant appetite for artifacts.   It isn’t a problem for Central, and I’m not writing this because Central is falling down on the job. Central can handle it, but it certainly isn’t the most efficient way to support millions developers.  It isn’t a good use of network bandwidth, and it isn’t a good use of energy to constantly cart around the same static JARs over and over and over again when the solution is so easy.

If everyone who used a build tool that interacted with Central adopted a repository manager such as Nexus we’d have a faster, more responsive system.   Central’s maintainers would be focused less on addressing the occasional runaway build and could spend more time and resource on increasing availability and functionality of this essential service.

Broken Builds

The other factor playing into this is that Maven builds only download releases once.   It isn’t like these build tools are repeatedly returning to Central to download release artifacts over and over again.

Well… actually… that isn’t true, we’ve seen some installations of Hudson configured to delete a local repository before every build placing a high load on Central.   Imagine a build that downloads 50 MB of dependencies running once every 5 minutes.  That’s one build consuming ~14 GB a day never mind the time wasted downloading static artifacts. While these broken builds are the exception, they do still show up from time to time.  Central can handle the load, but imagine 1000 of these broken builds running continuously and you can see the challenge.

A Simple Reminder: Please Proxy Central

We’re constantly watching the performance of the system and making sure it stays up and running for an entire world of developers.  If you use a build tool that hits Central whether it is Buildr or Maven or Gradle or Ivy, you can help us by running a Nexus instance.

Even if all of your builds work perfect, running a local Nexus instance helps preserve Central as a public, free resource and it will lead to faster, more responsive builds.

In this post, I focus on storage requirements for Nexus. I discuss general recommendations and point you at resources we’ve developed to help you come up with accurate estimates for how much disk space you’ll need.

Disk Space

Disk space is going to be the critical parameter for a Nexus installation. At its core, Nexus is simply a collection of files and a set of services to index and serve these files. If you integrate Nexus into your development process and come to depend on it as a collaboration mechanism, you can easily consume hundreds of gigabytes (or even terabytes) of space.

Coming up with a simple guideline for storage requirements is difficult as it depends on a number of factors: How many projects do you have? How large are the artifacts being deployed to Nexus? How frequently are these artifacts deployed? and How long do you keep your releases? How much open source are you consuming from Central? How often do you update external dependencies? and How many 3rd party artifacts do you need to upload?

If you deploy artifacts to Nexus, your internal, hosted repositories are what will end up consuming the most space over time. At a large organization with hundreds of projects and frequent releases, it is very easy to create systems that consume a surprising amount of space. If you are interested in diving into the details and coming up with an estimate for your organization, watch “Getting Scientific about Sizing Nexus”.

An Initial Starting Point

While some of our engineers like to aim high with an initial recommendation of 250-500 GB, I like to aim a little lower. Sure, if you are rolling Nexus out to a 5,000 developer installation with thousands of projects, you may very well want to start with 1 TB. On the other hand, if you are gradually rolling Nexus out to a department or two, you should start with a more reasonable number: 50 or 100 GB.

I recommend starting with 50 or 100 GB, and I also recommend being prepared to expand that number as needed. Starting with this smaller number avoids the problem of procuring a huge chunk of disk space only to watch it sit idle for the months (or years) it will take you to consume all this space. Aim low, plan to expand.

Conclusion

Your initial estimate for disk space consumption is going to be just that, an estimate. Having set up scores of Nexus instances for organizations of all sizes, my experience has been that you’ll want to do some ballpark estimates and then multiply that estimate by a factor of two or three. When you connect systems like Hudson to Nexus and deploy snapshots from every integration build, you’ll appreciate the extra space.

As you start to use Nexus, you’ll have to tweak your scheduled jobs to make sure that you are periodically removing old snapshots and regularly keeping an eye on storage. If you expand the number of projects or developers using a Nexus instance, you’ll want to revisit some of these initial estimates and make sure that your system has enough storage to keep track of all the artifacts it is caching and storing.

Sometimes Releases are Disposable

For example, consider a company that creates a large web-based system. They may deploy new versions of components to production multiple times a day. If this seems unrealistic, know that system administrators for popular services like Facebook, Last.fm, and Flickr have talked openly about how frequently code is pushed to production systems. A few times a day isn’t odd in some of these environments, and with a large system, you’d consume terabytes of space to keep all of those old releases around.

Last time I worked at a large consumer-focused web site, pushing something to production once a day wasn’t uncommon, and even the idea of rolling back to anything other than yesterday’s build was laughably impractical. If you identified a bug in a CMS or a production database, you’d just craft the fix it and move on. This is especially true of larger, web-facing systems in which the only reality is the code that is running in production today. A site like Flickr gains nothing from being able to roll back to a release from last September.

The Other Side of the Coin: Releases are Forever

Compare this to the production release schedule of a critical, supported product and it’s like night and day. If you are coding some serious banking system you might be lucky to have a release once a month. In all likelihood you might be looking at a quarterly release.  When you are working on critical applications, ship software, or have infrequent release cycles then the ability to roll back to previously released binaries is very important. When you work somewhere fast-paced with a very short release cycle, there’s not much value in retaining older releases.

For this reason, we’re clarifying this point: if it makes sense for you to delete release artifacts from a hosted repository, go for it. Just remember to re-index the server once you’ve manipulated the storage folder.

We put together the following video to share more of our thoughts on this topic.

• Integrated security and licensing reports to discover problematic components
• Improved availability with our new Nexus Availability Architecture
• Ability to manage both Java and .NET components from a single repository

The webinar recording is now available here.

Special pre-release pricing is available for Nexus Pro 2.0. Check out the webinar or contact a representative for more information.

Stop Running Tasks

You can stop running processes immediately and no longer have to wait for them to complete.  This is quite useful for recovering from high load situations.  Stopping a running process is easy — just select the process and click on the cancel button (see Figure 1). You can also delete tasks without waiting for them to finish. Delete will automatically stop the process and then delete it.

Improved Maven 2 Support

You’ll get improved support for environments with both Maven 2 and Maven 3 clients. Nexus now suppresses Maven 3 metadata automatically when communicating with Maven 2 clients. This improvement allows you to use such mixed environments successfully.

Enhanced User Information

Sonatype Pro for Nexus customers can better audit who uses each Nexus server with usage information that includes userid and user agent.

Download Nexus Today

So what are you waiting for, download the latest version of Nexus OSS  or the professional version from our support site.  Learn more about Sonatype Pro for Nexus.

Figure 1. Stop running processes easily with the new cancel button

But the good news is, with a Nexus repository manager you don’t need to worry about this ever again.

This video gives a quick overview of the benefits of using a Nexus to proxy Maven Central. It also goes over how to store third party libraries, and host artifacts.

Watch the video below:

The first blog in the series is by Sonatype software developer Juven Xu, on backing up Nexus.

Nexus is the industry leading repository manager that helps reduce build times and increase your control of open source artifacts by  managing software artifacts required for development, deployment, and provisioning. Nexus greatly simplifies the maintenance of your own internal repositories and access to external repositories such as Maven Central.  With Nexus you can completely control access to, and deployment of, every artifact in your organization from a single location.

If you are already using Nexus, this article will teach you how to back up your repository manager (if you don’t use Nexus, you can click here for more information).

Why Backup

To backup Nexus simply means to make a copy of your Nexus files for safekeeping. The copy should be stored on different hardware, other than the original one. For example, you might want to copy your Nexus settings in the sonatype-work/nexus/config folder to a removable hard disk.

It is important to backup your Nexus files because sometimes things fail. Hard disk might crash, files might be deleted by other programs, even ourselves might delete important files by mistake. With a backup, you will be protected from these frustrating events.

What To Backup

Nexus consists of two parts, one part is the runtime web application, which can always be downloaded from Sonatype site, the other part is the sonatype-work/ folder, which contains all the user data and configuration. So here our focus is the sonatype-work/ folder. If you are running the Nexus bundle, this folder is next to the nexus-webapp directory. If you are using Nexus war, this folder should be under the user home directory by default.

You can simply backup the whole sonatype-work/ directory if you are rich in disk space, but it’s not necessary. Some of the files in the directory are very important, or even unreproduceable, while some of the files can be easily regenerated. Here is a full list of sub-folders, from the most important to least:

1. nexus/storage (hosted repositories) — When you are using Nexus, you deploy lots of internal artifacts, which can’t be retrieved from public repositories, into these hosted repositories. Each repository has a corresponding folder in this directory. For example, hosted repository with id ‘release’ has a folder ‘nexus/storage/release’. Because the artifacts stored in hosted repositories are not likely hosted in any public repositories, we must backup them.
2. nexus/conf —  All the nexus configuration files are stored here, such as repository configuration, security configuration, and log configuration. Although you can recreate them in theory, it would be a pain to lose them — it takes time to recreate them, so, back up this folder.
3. nexus/logs —  All the nexus log are stored here, you might want to backup them to make sure you know the history.
4. nexus/timeline — Most of the important events like authentication failure, scheduled task starting, and recently deployed artifacts, are recorded and showed via RSS feeds. These events are internally stored in this folder. You can backup them for potential backtrace.
5. nexus/storage (proxy repositories) —  Once you’ve been using Nexus for a long time, huge amount of artifacts are cached from public repositories like Maven Central. The speed of your Maven builds benefits from these cached artifacts. Since these artifacts can be retrieved from public repositories at any time, there’s no need to backup them. But, if you have adequate disk space, you can still consider backup them, since this can save a lot of time for retrieving them again.
6. nexus/trash — Should you backup trash? Most of time these no need to do this. When artifacts are deleted in Nexus, they are actually moved to the trash, you can choose to backup them if you think you are likely to do stupid things very often.
7. nexus/indexer — Nexus indices are store in this folder, since Nexus can rebuild indices using the reindex task, there is no need to backup this folder.
8. nexus/proxy — Artifacts’ attributes like repository path, is readable, and last requested time are stored in this folder. Nexus can rebuild these attributes as well, so there is no need to backup them.

How To Backup

Now that you know what files to backup according to your situation, you need to know how to backup your files.

Take this example:

My Nexus is running on Ubuntu Linux, the path of sonatype-work directory is /home/juven/bin/sonatype-work/, I have a removable disk which is mounted at /media/disk/, I am using rsync to make backup:

$ rsync -a -delete -v ~/bin/sonatype-work /media/disk/

This command is equivalent to:

$ cp -a ~/bin/sonatype-work /media/disk/

except that it’s more efficient if there are only a few differences.

• The -a option tells rsync to run in archive mode, which means to run recursively, keep file mode, ownership, and keep symbolic link etc.
• The -delete option tells rsync to delete files in the target directory if they were deleted in the source directory.
• The -v option tells rsync to show verbose log.

So this command will backup the whole sonatype-work folder, what if I only want to backup some important the sub-folders?

Suppose I only want to backup these folders: /sonatype-work/nexus/storage/releases/, /sonatype-work/nexus/storage/thirdparty/, /sonatype-work/nexus/conf/, /sonatype-work/nexus/storage/logs/, and /sonatype-work/nexus/storage/timeline/, create a rsync includes list file like this:

+ /sonatype-work/nexus/storage/ + /sonatype-work/nexus/storage/releases/ + /sonatype-work/nexus/storage/thirdparty/ + /sonatype-work/nexus/conf/ + /sonatype-work/nexus/logs/ + /sonatype-work/nexus/timeline/ - /sonatype-work/nexus/storage/* - /sonatype-work/nexus/*

Automate The Backup

If your backup disk is always connected to your machine and you don’ t want to repeat the backup command manually, then schedule a task running the backup periodically.

On Linux, cron is a perfect tool for automating the backup task. In my case on Linux, with user name ‘juven’, edit file /etc/crontab, add a line like this:

0 0 * * 0 juven rsync -a -delete –include-from ~/bin/includes.list ~/bin/sonatype-work /media/disk/

This entry tells cron to run the rsync command as user juven, weekly.

And don’t forget to restart the cron deamon:

$ sudo /etc/init.d/cron restart

And that’s it! It should automatically perform the backup at the time you specified.