In addition to managing and maintaining the Maven Central repository, I also serve as the administrator for two large forge repositories: repository.apache.org. This post will dive into the best practices I've developed to maintain these large instances. I will focus on the configuration of Sonatype Nexus Repository in this post.
Both of these repositories have a few things in common that have driven the design:
There are many disparate projects deploying artifacts that require fine grained access control per project.
Release repositories are synced to central.
They are the most commonly used snapshot repositories in the maven ecosystem.
Most users anonymously read the snapshots.
They are transitional repositories that replace older static repositories.
They also have a few things that are different:
Apache is a Solaris Zone.
Codehaus is an Ubuntu Jeos VM.
Apache uses httpd for reverse-proxying and SSL.
Codehaus uses NGINX for reverse-proxying and SSL.
This post contains two sections, the first covers some system-wide Sonatype Nexus Repository configuration, the second details about adding individual projects, along with security and staging configuration. If you are setting up a public Maven repository, this post might give you some ideas about configuration and administration issues that you'll need to think about.
We want to protect all authenticated traffic, so both systems rewrite all http access to https (you can see how that's done in the server setup linked above). However, since 99% of all traffic to these systems is anonymous, I've allowed the snapshot URLs to poke through without being redirected.
Since I am using reverse proxies in front of Nexus, and the protocol doesn't have a good way to tell Nexus what the inbound protocol was, I need to tell Nexus how to generate absolute URLs used in the REST API. This is done by setting the following options in the server configuration pane.
The systems are configured with only two hosted repositories: releases and snapshots. Both systems are transitional, meaning that projects elect to convert at a convenient time. To support this, I proxy the old snapshot repository and aggregate it with the locally hosted snapshot repo. When you hit http://repository.apache.org/snapshots, you're hitting this group and it appears as one repository. We also have a staging group that is used to aggregate all staging repos that haven't yet been promoted.
One benefit to using Nexus in these forge setups is that we can configure rules that automatically check staged artifacts before they can be promoted. This includes things like validating the PGP signature is present and signed with a publicly accessible key, looking for sources and javadocs, validating the pom, etc. This is one way we are helping improve the data in Central, by helping correct it right at the source. Since these rules are tied to the Staging support, we want to disable the ability to deploy directly to the releases repository.
I also have configured the following jobs:
Configuration backup: Backs up the Nexus Configuration files. I have it set to run daily and keep 10 days of backups
Publish Indexes - This packages the internal real-time indexes into a format that is consumable by downstream Nexus' and M2eclipse users (other tools also consume this data). I have this set to run daily.
Purge proxy artifacts: Since we're transitional and proxying the old, static snapshot repositories, I have configured a task to evict items that haven't been requested in more than 10 days. This just reduces the disk consumption on the repo. If a file is re-requested later, it will be retrieved again from the proxy on demand.
Snapshot cleanup: We want to enforce best practices and keep snapshots moving forward. The cleanup task is set to keep a maximum of 3 timestamped snapshots for each artifact for a minimum of 10 days. All snapshots for an artifact are also purged when a release is promoted.
Empty the trash: All delete operations in Nexus never actually delete, they just move files to a trash folder. This is a security net if you misconfigure a cleanup task, or simply make a mistake in the UI (like dropping a repo you meant to promote). We keep on top of the trash by scheduling it to run once a week. New in 1.4 is the ability to purge things from the trash only after they have been deleted for x days. I've set this to hold things in the trash at least 7 days. This gives projects more than enough time to detect any issues and recover the artifacts.
Each project needs to have access to only their own artifacts. Nexus supports two different ways to handle the security separation. If you want to read more about the two modes, read my previous post: "Optimal Nexus Repository Configuration." We have chosen to manage the forge by partitioning a single pair of repositories. The next few steps show how this is done.
First, we define a new Repository Target for the artifacts of this project. Don't worry if you're not a regexp genius, wildcards are very easy. We let you define multiple regexps so you don't have to figure out more complicated and/or expressions. In the image below, I've created a new target called "org.codehaus.org" that will contain all artifacts in the paths /org/codehaus/mojo and below.
Now that we've defined our "bucket" of artifacts, we need to create some permissions associated with it. You're probably thinking "create permissions?" Yes, see the Repository Target is a generic concept that lets you arbitrarily group artifacts, but notice we haven't yet associated the target with any repositories.
NOTE: The logic behind this approach is that you may want to grant people read access to all org.foo artifacts, but what if you only want them to see artifacts that have been promoted and not things that are still being staged?
In this case, we want to grant CRUD (Create / Read / Update / Delegate) to SNAPSHOT artifacts, but only CRU to releases. (Admins are only allowed to delete releases, which prevents problems once things are synced to Central). I will do this in two steps, as shown below. First, create a set of permissions that link the org.codehaus.mojo Repository Target to "All Repositories".
Then create a set of permissions that only apply to the hosted Snapshot repository.
Both the Apache and Codehaus forges use the Staging support in conjunction with Staging rules to validate the integrity of releases.
NOTE: Nexus staging is unique in that it's entirely controlled from the server side, which means Admins can adjust as needed without changing the poms. It's also designed so that all projects use a single URL for deployment that is abstracted from the repository, which provides two benefits: 1) you can change the repository hosting artifacts without changing poms and 2) you can specify the distributionManagement URL in just one place, reducing the errors. For example, at Apache we have a parent pom that contains all the logic a project needs to be staged.
We control this in Nexus by creating a Staging Profile. Fortunately, the profile reuses the Repository Target we defined earlier.
Not shown are the settings that let you set the validation rules, and who should be notified at each promotion step. Now that we have defined the staging profile, the system automatically created a few new permissions that let you specify who is allowed to stage, drop and promote these artifacts. We want to grant these permissions to users, and this is done via roles.
The Codehaus repository is linked to their LDAP system. Nexus takes a unique approach that allows you to easily grant access to dozens of users without having to configure each user in the system. We do this by allowing you to "map an external role" and then grant Nexus specific permissions to any user with this role. To do this, navigate to the Role pane and select the Add External Role Mapping as shown below.
This opens a dialog where you can select the external realm (you could have multiple realms), and then you will see a list of all the roles known to the external system. Here I'm selecting "mojo-developers."
This creates a new role in Nexus, with an id matching the external system id "mojo-developers." Any user authenticated with this role in the external user account will automatically be granted these permissions.
I now select the roles and permissions I want to grant. Specifically, I grant the following:
Staging: Deployer (org.codehaus.mojo) - This is a role created when I setup the profile. It contains the basic permissions needed to allow a user to view, stage, close and drop org.codehaus.mojo staging repositories (only)
UI: Staging Repositories - This lets the user actually see the staging view, without this they couldn't see what they staged.
org.codehaus.mojo - All - Here I'm granting Create, Read and Update for all matching artifacts (remember, these are the permissions we created above that apply to all repositories)
org.codehaus.mojo - snapshots: Here I'm granting delete, but only for the org/codehaus/mojo artifacts in the Snapshot repository.
Staging: Profile org.codehaus.mojo - (promote): The Staging: Deployer xxx roles give the ability to stage, but not the ability to promote. This permission may be granted to managers, QA, or PMC etc. as appropriate. Here we're letting developers stage and promote their own artifacts.
And we're done. Notice that I didn't need to grant permissions to every user, and I didn't have to put the users into groups. This is the power Nexus provides in user management. This is core functionality that applies to any realm you may have, not just Sonatype Nexus Repository ones.
Now, to illustrate some more power of this approach, what if org.codehaus.mojo later starts managing artifacts under org.mojo? I don't have to redo everything here, I just extend the Repository Target "bucket" by adding ".*/org/mojo/.*" and instantly all the permissions, staging profiles, etc apply to the new groupId. This has definitely saved me many times at Apache with the Webservices Projects... they have more groupIds than I can count, but they all map back to the same external role. Each time a new one comes along, I just add it to the target and I'm done.
This is the process we've ironed out over several months. I definitely don't go through this UI clicking every time. Since Nexus has a full REST API (the UI is just a REST client built in JavaScript), we have developed a set of command line tools that take a few simple inputs like groupId, external role id and project name, and automate all this configuration via REST calls automatically.