We are moving both our public and private source repositories to Git for a few reasons:

• Git works better for people with slow or lagging connections
• Branch management is easy
• Submitting patches via a pull request is easier to deal with than a patch attached to a bug report

We started out with SVN mostly because that was what everyone was used to. A year ago or so, one of our remote developers started using a git-svn mirror to remove some of the latency issues they had with SVN. Someone else put together a ten page wiki on how to easily use a git-svn mirror and push back to the canonical SVN repo. As time went on, interest in git grew and new modules were created in git, instead of our SVN repositories.  Finally there was a push to move everything to git.

There were a couple different directory conventions used for our repositories.  Our Nexus SVN repository used the typical project-root/trunk,tags,branches, and git-svn worked nice right out of the box. Our other repositories lumped all modules under the trunk, tags, and branches folder (figure 1).

We decided to give each module its own git repository to better manage tags and branches related to the given module. This made the git import a pain, as you cannot just export tags/* (and tags/module-B-* will not work either), you need to be more selective and configure each tag to pull.

Figure 1.

In a nut shell you need to:

(1) Create a new git repository.

$ git svn init –trunk=trunk/project <svn-root> <repo-name>

(2) Configure the specific tags you want to export (we use the Maven release plugin, so the tagging convention is the same)

$ git config –add svn-remote.svn.tags tags/{module-B-2.1, module-B-2.2}:refs/remotes/tags/*

(3) Fetch everything (for an automated way of generating an authors file see this blog post)

$ git svn fetch –authors-file=<svn-authors-file>

(4) Next you need to clean up the tags and push.

If you script this, you end up with something like:

#!/bin/bash

CURRENT_PATH=`pwd`

# file based URL for reads
SVN_URL=file:///path/to/repositories/sonatype.org/spice
SVN_REMOTE_URL=https://svn.sonatype.org/spice
SVN_TAGS_DIR=tags
SVN_TAG_PREFIX=$1
SVN_TRUNK_DIR=trunk/${SVN_TAG_PREFIX}
SVN_AUTHORS=/path/to/svn-authors.txt

GIT_REPO_NAME=$1
GITHUB_USER=userid
GITHUB_TOKEN=xxx
GITHUB_TEAM_ID=1234
GITHUB_ORG=sonatype

# this is the magic, get the list of tags that match this project the output format looks like:
# tags/{module-1.0,module-1.2,module-2.1}
GIT_TAG_ARG=${SVN_TAGS_DIR}/\{`svn list ${SVN_URL}/${SVN_TAGS_DIR} | grep ${SVN_TAG_PREFIX} | sed 's_/__' | xargs -I {} echo -n ,{} | sed 's_^,__'`\}

#echo ${GIT_TAG_ARG}

# init git repo
git svn init --trunk=${SVN_TRUNK_DIR} ${SVN_URL} ${SVN_TAG_PREFIX}

#copy authors text to repo
cp ${SVN_AUTHORS} ${SVN_TAG_PREFIX}/.svn-authors.txt

# move to directory
cd ${SVN_TAG_PREFIX}

# make sure we have tags for this project
if [ `svn list ${SVN_URL}/${SVN_TAGS_DIR} | grep ${SVN_TAG_PREFIX} | wc -l` -gt 0 ]; then
  #configure the tags, we only want a subset of the tags, because we lumpped all the tags in a single place.
  git config --add svn-remote.svn.tags ${GIT_TAG_ARG}:refs/remotes/tags/*
fi

#fetch
git svn fetch --authors-file=.svn-authors.txt
# make sure there are no errors before we continue
if [ "$?" -ne "0" ]; then
  echo "Error while fetching, see above"
  exit 1
fi

git update-ref -d master

##########################################################
# fix the tags, from: https://github.com/nothingmuch/git-svn-abandon
##########################################################

# create annotated tags out of svn tags
git for-each-ref --format='%(refname)' refs/remotes/tags/* | while read tag_ref; do
    tag=${tag_ref#refs/remotes/tags/}
    tree=$( git rev-parse "$tag_ref": )

    # find the oldest ancestor for which the tree is the same
    parent_ref="$tag_ref";
    while [ $( git rev-parse --quiet --verify "$parent_ref"^: ) = "$tree" ]; do
        parent_ref="$parent_ref"^
    done
    parent=$( git rev-parse "$parent_ref" );

    # if this ancestor is in trunk then we can just tag it
    # otherwise the tag has diverged from trunk and it's actually more like a
    # branch than a tag
    merge=$( git merge-base "refs/remotes/trunk" $parent );
    if [ "$merge" = "$parent" ]; then
        target_ref=$parent
    else
        echo "tag has diverged: $tag"
        target_ref="$tag_ref"
    fi

    # create an annotated tag based on the last commit in the tag, and delete the "branchy" ref for the tag
    git show -s --pretty='format:%s%n%n%b' "$tag_ref" | \
    perl -ne 'next if /^git-svn-id:/; $s++, next if /^\s*r\d+\@.*:.*\|/; s/^ // if $s; print' | \
    env GIT_COMMITTER_NAME="$(  git show -s --pretty='format:%an' "$tag_ref" )" \
        GIT_COMMITTER_EMAIL="$( git show -s --pretty='format:%ae' "$tag_ref" )" \
        GIT_COMMITTER_DATE="$(  git show -s --pretty='format:%ad' "$tag_ref" )" \
        git tag -a -F - "$tag" "$target_ref"

    git update-ref -d "$tag_ref"
done

# create local branches out of svn branches
git for-each-ref --format='%(refname)' refs/remotes/ | while read branch_ref; do
    branch=${branch_ref#refs/remotes/}
    git branch "$branch" "$branch_ref"
    git update-ref -d "$branch_ref"
done

# remove merged branches
git for-each-ref --format='%(refname)' refs/heads | while read branch; do
    git rev-parse --quiet --verify "$branch" || continue # make sure it still exists
    git symbolic-ref HEAD "$branch"
    git branch -d $( git branch --merged | grep -v '^\*' )
done
##########################################################
# done fixing tags
##########################################################

# we should already be on the master, but lets make sure
git checkout master

##
# Use the Github API to create a repo and add it to the team.
##

#create github repo
curl -F "login=${GITHUB_USER}" -F "token=${GITHUB_TOKEN}" https://github.com/api/v2/json/repos/create -F "name=${GITHUB_ORG}/${GIT_REPO_NAME}" -F "has_issues=false" -F "has_downloads=false" -F "has_wiki=false"  --request POST

#add the dev team to the repo
curl -F "login=${GITHUB_USER}" -F "token=${GITHUB_TOKEN}" -F "name=${GITHUB_ORG}/${GIT_REPO_NAME}" http://github.com/api/v2/json/teams/${GITHUB_TEAM_ID}/repositories

# set the origin
git remote add origin git@github.com:${GITHUB_ORG}/${GIT_REPO_NAME}.git
#push it all
git push --tags origin master

#go back to where we started
cd ${CURRENT_PATH}

# some reminders so we do not forget the manual steps
echo "--${GIT_REPO_NAME}--" >> migrate.out
echo "You should now review the git repo at: https://github.com/${GITHUB_ORG}/${GIT_REPO_NAME} then remove the svn repo with:" >> migrate.out
echo "$ svn rm ${SVN_REMOTE_URL}/${SVN_TRUNK_DIR} -m  \"Moved to github: https://github.com/${GITHUB_ORG}/${GIT_REPO_NAME}\"" >> migrate.out
echo "Do not forget to update the grid: https://grid.sonatype.org/ci/\" to use: git://github.com/${GITHUB_ORG}/${GIT_REPO_NAME}.git >> migrate.out
echo "Add redirect from ${SVN_REMOTE_URL}/${SVN_TRUNK_DIR} to https://github.com/${GITHUB_ORG}/${GIT_REPO_NAME}" >> migrate.out
echo "" >> migrate.out

For my talk today at JFokus today I’ve taken the liberty of starting some notes for folks interested in attending. There’s a lot to cover and so I thought I would try the approach of providing some material up front so the session can be more of a dialog. I’m going to attempt to cover everything in the picture below and save the demos folks might want to see for the Sonatype booth. Happy to chat with folks and do any demos before and after the presentation. Just stop by!

Maven Stack Infrastructure

I’m going to talk about some of the under pinnings of the technologies we’re using as part of our Maven work. Why we selected the technologes and some of the current work that’s happening.

• Guice
• From Plexus to Guice (#1): Why Guice?
• From Plexus to Guice (#2): The Guice/Plexus Bridge and Custom Bean Injection
• From Plexus to Guice (#3): Creating a Guice Bean Extension Layer
• Peaberry
• Apache Shiro
• Enunciate

Maven

There are several develops going on in the various Maven projects. Maven 3.x is on the way, but we have some very interesting work happing with OSGi within our Tycho project. The Flexmojos project and NAR (the C/++ framework for Maven) are also popular.

Tycho

• Tycho Site
• Tycho @ Github
• Tycho/PDE Integration
• P2 Site

Flexmojos

• Flexmojos Site

NAR

• NAR Site
• NAR @ Github

M2Eclipse

The primary IDE integration we work on at Sonatype. The most important thing to talk about in M2Eclipse is the configuration framework.

• M2Eclipse Site

Hudson

Most of the work we’ve been doing on Hudson is still not really good enough for public consumption but we’re testing the changes we’re making on the Sonatype grid.

Hudson Site

Nexus

• Nexus Site
• Nexus: Improving Maven Central and Supporting the Maven Ecosystem
• Selecting OSS Software: 10 Questions Answered for Sonatype Nexus
• OSS Repository Hosting
• OSGi Central
• Ruby Central (We’re working on making that publicly available

Proviso

Proviso is not publicly released yet, but Alin and I have been working together to get our provisioning framework ready for a public release.

Git

We are starting to use Git heavily and soon likely exclusively. We are helping out on the EGit, and JGit projects at Eclipse and we’re trying to put a little Git server together based on JGit, MINA, and Apache Shiro.

• JGit Site
• EGit Site
• Tycho building EGit/JGit
• SShJGit @ Github

Maven Extensions

I’ll chat about these in the talk.

• Polyglot Maven Site
• Polyglot Maven @ Github
• Maven Shell
• Maven Shell @ Github

I’ll likely keep adding to the entry leading up the talk, but I thought I would get the ball rolling!

Sonatype uses GitHub to host a number of projects including all of our books. It has been a very valuable tool for us, and we’ve already seen great benefits. The social, interactive nature of the tool allows people interested in the book to keep up with the development of the content, and we’ve already had a few contributors show up and help us write more content. Contributors can fork our books, add whole sections or correct typod and then make a simple pull request that notifies us of the changes. Because we find it so useful, I thought I’d take some time to detail the process of connecting a new Maven project to GitHub. The following post details the process of creating a new GitHub repository, importing a Maven project, and then configuring the SCM element in your project’s POM. Once you’ve done that, you can start using the Maven Release plugin to automate the SCM operations that accompany a release.

Step 1. Install Git on your local machine. If you don’t know how to do this, don’t worry, GitHub is an example of a site that values great documentation, and they have a number of helpful guides for Windows, Mac OS X, and Linux.

Step 2. Sign up for a GitHub account if you don’t already have one.

Step 3. Once you create an account, you’ll need to load your public SSH key into GitHub. When you run git, or more importantly when Maven runs Git, it isn’t going to prompt you for a username and password, it relies upon key-based authentication in SSH. Again, if you don’t know how to do this, follow one of these GitHub help documents: Windows, Mac OS X, and Linux.

To add a public key (you can have several associated with your account), login, click on “account”, and then click on “add another public key” under the section that lists public keys.

Step 4. If you don’t already have Maven installed, download it from http://maven.apache.org and add it to your PATH. If you are looking for more detailed instructions, we have an entire chapter on the Maven installation process in Maven: The Definitive Guide.

If you can run “mvn” and “git” from the command line, you are ready to start creating a project and importing it into a new GitHub repository.

Step 5. Create a new Maven project using the archetype:generate goal. Run “mvn archetype:generate” from the command line, supply a groupId, artifactId, package name, version, and use the default quickstart Archetype.

For the purposes of this blog entry, I’ll assume you used the following parameters:

• groupId: com.sonatype.blog
• artifactId: git-demo
• version: 1.0-SNAPSHOT
• package name: com.sonatype.blog

Step 6. Change directories into the newly created “git-demo” directory and run “mvn clean install” to make sure that you can successfully build the quickstart project.

Step 7. Create a new GitHub repository. Login, and click on the GitHub logo in the upper left-hand of the web page, you should then see a list of your repositories on the right-hand side of the screen. Click on “Create a New One” next to the words “Your Repositories” (as shown below)

For the purposes of this post, I used the following values:

Once your new GitHub repository is created, GitHub will load the repository and show you some instructions. At this point, you’ll want to import the project you just created into your new GitHub repository.

Step 8. From the “git-demo” project directory, execute the following commands (substitute your Git user name for “tobrien” in the commands listed below):

   mvn clean
   git init
   git add -A
   git commit -m "Importing the git-demo project"
   git remote add origin git@github.com:tobrien/git-demo.git
   git push origin master

After running these commands, return to the GitHub interface, and you should see that your GitHub project is now full of the contents of the git-demo project. While you’ve added the project to GitHub, you still haven’t configured your Maven project to interact with GitHub, let’s do that now….

Step 9. Open up the pom.xml file of the git-demo project and add the following scm element right before the closing project element. Make sure to substitute your account name for the instances of “tobrien” in the XML snippet shown below:

 <scm>
	<connection>scm:git:git@github.com:tobrien/git-demo.git</connection>
	<url>scm:git:git@github.com:tobrien/git-demo.git</url>
	<developerConnection>scm:git:git@github.com:tobrien/git-demo.git</developerConnection>
  </scm>

After the scm element, add the following build element which forces Maven to use the most recent version of the maven-release-plugin:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-release-plugin</artifactId>
        <version>2.0-beta-9</version>
      </plugin>
    </plugins>
  </build>

Add the pom.xml to the next commit, commit your changes, and push to GitHub:

  git add pom.xml
  git commit -m "Configuring POM for GitHub project"
  git push

Step 10. Run the Maven Release Plugin to cut the 1.0 release. Since our quickstart project is at version 1.0-SNAPSHOT, running “mvn release:prepare” is going to update the version to 1.0, commit the changes to a git-demo-1.0 tag, and then increment the version to 1.1-SNAPSHOT. Do to this, run “mvn release:prepare”.

canning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building git-demo
[INFO]    task-segment: [release:prepare] (aggregator-style)
[INFO] ------------------------------------------------------------------------
[INFO] [release:prepare {execution: default-cli}]
[INFO] Resuming release from phase 'scm-check-modifications'
[INFO] Verifying that there are no local modifications...
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git status
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] nothing added to commit but untracked files present (use "git add" to track)
[INFO] Checking dependencies and plugins for snapshots ...
What is the release version for "git-demo"? (com.sonatype.demo:git-demo) 1.0: :
What is SCM release tag or label for "git-demo"? (com.sonatype.demo:git-demo) git-demo-1.0: :
What is the new development version for "git-demo"? (com.sonatype.demo:git-demo) 1.1-SNAPSHOT: :
[INFO] Transforming 'git-demo'...
[INFO] Not generating release POMs
[INFO] Executing goals 'clean verify'...
[WARNING] Maven will be executed in interactive mode, but no input stream has been configured for this MavenInvoker instance.
[INFO] [INFO] Scanning for projects...
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] Building git-demo
[INFO] [INFO]    task-segment: [clean, verify]
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] [clean:clean {execution: default-clean}]
[INFO] [INFO] [resources:resources {execution: default-resources}]
[INFO] [WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] [INFO] skip non existing resourceDirectory /Users/tobrien/temp/git-demo/src/main/resources
[INFO] [INFO] [compiler:compile {execution: default-compile}]
[INFO] [INFO] Compiling 1 source file to /Users/tobrien/temp/git-demo/target/classes
[INFO] [INFO] [resources:testResources {execution: default-testResources}]
[INFO] [WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] [INFO] skip non existing resourceDirectory /Users/tobrien/temp/git-demo/src/test/resources
[INFO] [INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] [INFO] Compiling 1 source file to /Users/tobrien/temp/git-demo/target/test-classes
[INFO] [INFO] [surefire:test {execution: default-test}]
[INFO] [INFO] Surefire report directory: /Users/tobrien/temp/git-demo/target/surefire-reports
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.sonatype.demo.AppTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec
[INFO]
[INFO] Results :
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] [INFO] [jar:jar {execution: default-jar}]
[INFO] [INFO] Building jar: /Users/tobrien/temp/git-demo/target/git-demo-1.0.jar
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] BUILD SUCCESSFUL
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] Total time: 5 seconds
[INFO] [INFO] Finished at: Tue Sep 08 17:35:35 CDT 2009
[INFO] [INFO] Final Memory: 24M/80M
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] Checking in modified POMs...
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git add pom.xml
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git status
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git commit --verbose -F /var/folders/qR/qRpEDfQcFPmRZpsXgPHOok+++TM/-Tmp-/maven-scm-2142457355.commit pom.xml
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git push
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Tagging release with the label git-demo-1.0...
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git tag -F /var/folders/qR/qRpEDfQcFPmRZpsXgPHOok+++TM/-Tmp-/maven-scm-1032558296.commit git-demo-1.0
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git push origin git-demo-1.0
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git ls-files
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Transforming 'git-demo'...
[INFO] Not removing release POMs
[INFO] Checking in modified POMs...
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git add pom.xml
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git status
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git commit --verbose -F /var/folders/qR/qRpEDfQcFPmRZpsXgPHOok+++TM/-Tmp-/maven-scm-1856882176.commit pom.xml
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Executing: /bin/sh -c cd /Users/tobrien/temp/git-demo && git push
[INFO] Working directory: /Users/tobrien/temp/git-demo
[INFO] Release preparation complete.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24 seconds
[INFO] Finished at: Tue Sep 08 17:35:45 CDT 2009
[INFO] Final Memory: 14M/80M
[INFO] ------------------------------------------------------------------------

Step 10. After a release:prepare, the logical next step is to run a release:perform. release:perform exports the newly created git-demo-1.0 tag, builds the project and then calls deploy. If you run “release:perform” without a distributionManagement section of your POM you are going to get an error, but run it anyway just to see what the Maven Release Plugin does.

The next step on your journey involves the configuration of the distirbution management settings and the deployment of your project’s artifacts to a repository manager. Once you do that, you can completely automate the process of releasing, updating source control, and distributing your binary artifacts to a repository manager.