Scala Artifacts Now on Central
Two weeks ago, all Scala projects required a little bit of extra configuration to point to a custom repository for Scala artifacts hosted at scala-tools.org. Today, Scala artifacts are now available directly from Central. The contents of scala-tools.org are now integrated into the Sonatype OSS repository hosting service, and other projects have started to publish artifacts Central.
The Scala community will see immediate benefits from this move. There are no more extra repositories to configure. It just got incrementally easier to use Scala. If you are new to Scala, you don’t need to reconfigure your repository manager to proxy another remote repository. The community will benefit from Sonatype’s continued investment in the infrastructure that runs Central: a cluster of machines in both the US and the EU continuously monitored by a dynamic DNS server that can reroute traffic instantly in the event of downtime.
How did this happen? Joshua Suereth, David Bernard, and Derek Chen-Becker provided the bulk of the administrative work, and they recently decided to decommision this server and transition repository hosting to the free Sonatype OSS service. Here’s the announcement by Joshua Suereth to the user forums on scala-lang.org on January 17th:
Scala-tools.org is going down and not accepting any new OSS projects. For those of us who wish to continue release software, I recommend migrating over to Sonatype. They put a few (good practice) limitations on contributions, but scala-tools.org would have done the same before long anyway. The benefit of Sonatype hosting is that your projects will make it onto the maven-central repository and benefit from the myriads of mirrors. Here’s the link for how to get started contacting Sonatype: http://nexus.sonatype.org/oss-repository-hosting.html
Publishing Your Scala Project to Central via Sonatype OSS
If you maintain a project that previously published to the scala-tools.org repository, here are three resources that provide guidance for Scala developers looking to publish Scala artifacts to Central:
- Publishing artifacts to Sonatype instruction written by Joshua Suereth on publishing to Sonatype OSS
- Publishing SBT Projects to Sonatype OSS from Cake Solution’s Specs2 Spring Project
- PublishToSonatype.scala some Scala code from Paul Phillips to automate the process of publishing artifacts to Sonatype’s OSS
A Simple Reminder for Maven/Gradle/Ivy Users: Proxy Central
Over the course of the past few years, I’ve interacted with hundreds of people when talking about build tools and repository management. It continues to surprise me how many people don’t realize where these artifacts come from. When you run a build and these JARs just show up alongside all of their dependencies, it’s like magic to most people. If you know how it works, it’s very obvious to you that running a repository manager is the right thing to do. This post is a reminder to everyone using build tools that rely on Central: take time to proxy Central with a repository manager.
“Wait, that’s how Central works?”
There’s something so automatic about dependency management in Maven that it often takes people a few months to understand exactly where those JAR files are coming from. In an 8 hour Maven class, I get to dependencies in the third hour, and after describing Central, what it is was like before Central, how metadata is stored in a repository alongside binaries, transitive dependencies, etc…. it all falls into place, and people realize that this simple thing they’ve grown accustomed to is only easy because of a ten year effort to refine the model, the creation of a support structure for source forges at places like Oracle and Google, and a constant investment in infrastructure.
On the one hand, it’s a great success that Central is, for the most part, an invisible utility that supports developers. On the other hand, it’s the kind of thing that people can start to take for granted very easily.
For example, a few months ago I spoke to someone who worked in an environment disconnected from the internet for security reasons. This individual was talking about how limiting it was to have to download JARs from open source projects manually and assemble them in a project. His words were: “It’s like programming Java in 2001 again.”
How can you help?
Imagine millions of developer spread all over the world: different time zones, different applications, but they all hit the same service: Central. Some regions have more developers than others so we certainly see peaks in usage throughout the day, but in general, Central’s serving thousands of files throughout the world at any given time during the day.
Maybe someone just installed Maven for the first time, or maybe they blew away a local repository, with numbers like these we see a world that has a constant appetite for artifacts. It isn’t a problem for Central, and I’m not writing this because Central is falling down on the job. Central can handle it, but it certainly isn’t the most efficient way to support millions developers. It isn’t a good use of network bandwidth, and it isn’t a good use of energy to constantly cart around the same static JARs over and over and over again when the solution is so easy.
If everyone who used a build tool that interacted with Central adopted a repository manager such as Nexus we’d have a faster, more responsive system. Central’s maintainers would be focused less on addressing the occasional runaway build and could spend more time and resource on increasing availability and functionality of this essential service.
Broken Builds
The other factor playing into this is that Maven builds only download releases once. It isn’t like these build tools are repeatedly returning to Central to download release artifacts over and over again.
Well… actually… that isn’t true, we’ve seen some installations of Hudson configured to delete a local repository before every build placing a high load on Central. Imagine a build that downloads 50 MB of dependencies running once every 5 minutes. That’s one build consuming ~14 GB a day never mind the time wasted downloading static artifacts. While these broken builds are the exception, they do still show up from time to time. Central can handle the load, but imagine 1000 of these broken builds running continuously and you can see the challenge.
A Simple Reminder: Please Proxy Central
We’re constantly watching the performance of the system and making sure it stays up and running for an entire world of developers. If you use a build tool that hits Central whether it is Buildr or Maven or Gradle or Ivy, you can help us by running a Nexus instance.
Even if all of your builds work perfect, running a local Nexus instance helps preserve Central as a public, free resource and it will lead to faster, more responsive builds.
What is Central?
“Central”, “Maven Central”, “The Central Repository”. You’ll here these terms a lot when discussing Java open source-based development. At Sonatype, we often take it for granted that everyone knows what we mean when we say “Central”. We know that’s not true, so we’ve put together this short video overview of Central and what it means to the Java community. Enjoy. (more…)
Publishing Your Artifacts to the Central Repository
Sonatype makes it easy to add your projects to the Central Repository with a free, public hosting service called OSSRH. We first blogged about this back in 2009, but given the growth in the community, we thought some of you may not have seen that post, so we decided to update it. (more…)
Maven Indexer: Sonatype's Donation to Repository Search
We create a search index for the Maven repository so that you don’t have to. What does this mean for you? It means that you don’t have to run a “little Google” in your datacenter just to search for the latest log4j library, and you also don’t have to sacrifice Terabytes of bandwidth to download thousands of artifacts you’ll never use to just to find the handful you need for your project. This is all done for you on Central, and the tools you use to search Central, Nexus and m2eclipse all benefit from this pre-made index file.
While this seems like such a simple idea, the Maven ecosystem hasn’t had a standard way to search the repository for the majority of its history. For much of the last decade there was no reliable way to search for an artifact. In this post, I’m going to review this history and talk about Maven repository search and where we think search is headed. With the release of Nexus OSS 1.9 it is now a good time to summarize the results of Sonatype donation of the Nexus Indexer to the Apache Software Foundation.
