Sonatype Blog » indexer

Nexus Indexer API: Part 2

Damian Bradicich — Sun, 07 Jun 2009 16:16:23 +0000

This series of Nexus Indexer posts will focus on integrating the Nexus Indexer into your own application. If you have an application that needs to search for an artifact by GAV coordinates, or by class name, you can use the Nexus Index format and the Nexus Indexer API to very easily search and locate artifacts in any repository that creates a Nexus Index. There are four main functions that are exposed in the Nexus Indexer API.

Indexing (Part 1)
Searching
Packing
Updating

This post will focus on Searching and provide you with some concrete code examples that show you how to use the indexer search capabilities in your application.

Searching

The Nexus Indexer uses Apache Lucene to handle storage and searching of the index data, and provides a very simple method to search against fields in the index. Below I have extended my plexus component from the Part 1 of this blog series, to include search functionality.

package org.damian;
...
/**
 * Sample app to show how to integrate with the nexus indexer.  Note that this is a simple plexus
 * component extending the SampleApp interface
 *
 * public interface SampleApp
 * {
 *    void index()
 *        throws IOException;
 *
 *    Set searchIndexFlat( String field, String value )
 *        throws IOException;
 *
 *    Set searchIndexFlat( Query query )
 *        throws IOException;
 *
 *    Map searchIndexGrouped( String field, String value )
 *        throws IOException;
 *
 *    Map searchIndexGrouped( String field, String value, Grouping grouping )
 *        throws IOException;
 *
 *    Map searchIndexGrouped( Query q, Grouping grouping )
 *        throws IOException;
 * }
 *
 * @author Damian
 *
 */
@Component( role = SampleApp.class )
public class DefaultSampleApp
    implements SampleApp,
        Initializable,
        Disposable
{
    // The nexus indexer
    @Requirement
    private NexusIndexer indexer;
 
    ...
 
    // search for artifacts
    public Set searchIndexFlat( String field, String value )
        throws IOException
    {
        // Build a query that will search the documents for the field set to the supplied value
        // This uses predefined logic to define the query
        // See http://svn.sonatype.org/nexus/trunk/nexus-indexer/src/main/java/org/sonatype/nexus/index/DefaultQueryCreator.java
        // for details
        Query query = indexer.constructQuery( field, value );
 
        return searchIndexFlat( query );
    }
 
    // search for artifacts using pre-built query
    public Set searchIndexFlat( Query query )
        throws IOException
    {
        // Build the request
        FlatSearchRequest request = new FlatSearchRequest( query );
 
        // Perform the search
        FlatSearchResponse response = indexer.searchFlat( request );
 
        // Return the artifact info objects
        return response.getResults();
    }
 
    public Map searchIndexGrouped( String field, String value )
        throws IOException
    {
        // We will simply use the GAV grouping, meaning that each groupId/artifactId/version/classifier
        // will have its own entry in the returned map
        return searchIndexGrouped( field, value, new GAVGrouping() );
    }
 
    public Map searchIndexGrouped( String field, String value, Grouping grouping )
        throws IOException
    {
        // Build a query that will search the documents for the field set to the supplied value
        // This uses predefined logic to define the query
        // See http://svn.sonatype.org/nexus/trunk/nexus-indexer/src/main/java/org/sonatype/nexus/index/DefaultQueryCreator.java
        // for details
        Query query = indexer.constructQuery( field, value );
 
        return searchIndexGrouped( query, grouping );
    }
 
    public Map searchIndexGrouped( Query q, Grouping grouping )
        throws IOException
    {
        GroupedSearchRequest request = new GroupedSearchRequest( q, grouping );
 
        GroupedSearchResponse response = indexer.searchGrouped( request );
 
        return response.getResults();
    }
}

As you can see, there are 2 ways to handle how the search is performed. The first is to use the nexus indexer to decide what type of lucene Query to create, based upon the input string. The second is to pass in your own lucene Query object, you can use any object that extends the org.apache.lucene.search.Query abstract class. If you choose to let the nexus indexer build your query, the following rules will apply to the search term:

string converted to lowercase
if first character is ‘^’ drop it
if first character anything other than ‘^’ prepend string with ‘*’
if last character is ‘ ‘, ‘<’ or ‘$’ drop it
if last character anything other than ‘ ‘, ‘<’ or ‘$’ append ‘*’ to end of string
if modified string does not contain ‘*’ then a TermQuery is used (exact match)
if modified string contains ‘*’ in last position, then a PrefixQuery is used (and ‘*’ is dropped from string)
if modified string contains ‘*’ in any other position, then a WildcardQuery is used

There will probably be occassions where this list of rules doesn’t apply to your use case, so you instead will want to give the Indexer your own Query object, that leaves you to choose from the many Query objects that lucene provides, or even to make your own if necessary.

You will also notice that there are 2 different types of result sets you can get back, flat or grouped. Flat results are simply a Set of ArtifactInfo objects for each match in the index. Grouped results will combine results based upon the Grouping you request (in my sample, the default is GAVGrouping which will group items together that share the same groupId/artifactId/version/classifier).

Here are some samples from a unit test in the project source code, that do the different kinds of searching.

    public void testSampleSearch()
        throws Exception
    {
        app.index();
 
        Set artifacts = app.searchIndexFlat( SampleIndexCreator.MY_FIELD, "value" );
 
        assertNotNull( "returned artifacts is null", artifacts );
        assertFalse( "returned artifacts is empty", artifacts.isEmpty() );
 
        for ( ArtifactInfo ai : artifacts )
        {
            assertEquals( "returned artifact has invalid data", "value", ai.getAttributes().get( SampleIndexCreator.MY_FIELD ) );
        }
    }
 
    public void testSampleSearchWithTermQuery()
        throws Exception
    {
        app.index();
 
        // This type of query will be totally built outside of nexus indexer, and will not
        // be tied to constraints defined in
        // http://svn.sonatype.org/nexus/trunk/nexus-indexer/src/main/java/org/sonatype/nexus/index/DefaultQueryCreator.java
 
        // A TermQuery matches equal strings
        Query q = new TermQuery( new Term( SampleIndexCreator.MY_FIELD, "value" ) );
 
        Set artifacts = app.searchIndexFlat( q );
 
        assertNotNull( "returned artifacts is null", artifacts );
        assertFalse( "returned artifacts is empty", artifacts.isEmpty() );
 
        for ( ArtifactInfo ai : artifacts )
        {
            assertEquals( "returned artifact has invalid data", "value", ai.getAttributes().get( SampleIndexCreator.MY_FIELD ) );
        }
    }
 
    public void testSampleSearchWithPrefixQuery()
        throws Exception
    {
        app.index();
 
        // This type of query will be totally built outside of nexus indexer, and will not
        // be tied to constraints defined in
        // http://svn.sonatype.org/nexus/trunk/nexus-indexer/src/main/java/org/sonatype/nexus/index/DefaultQueryCreator.java
 
        // A PrefixQuery will look for any documents containing the MY_FIELD term that starts with val
        Query q = new PrefixQuery( new Term( SampleIndexCreator.MY_FIELD, "val" ) );
 
        Set artifacts = app.searchIndexFlat( q );
 
        assertNotNull( "returned artifacts is null", artifacts );
        assertFalse( "returned artifacts is empty", artifacts.isEmpty() );
 
        for ( ArtifactInfo ai : artifacts )
        {
            assertEquals( "returned artifact has invalid data", "value", ai.getAttributes().get( SampleIndexCreator.MY_FIELD ) );
        }
    }
 
    public void testSampleSearchWithWildcardQuery()
        throws Exception
    {
        app.index();
 
        // This type of query will be totally built outside of nexus indexer, and will not
        // be tied to constraints defined in
        // http://svn.sonatype.org/nexus/trunk/nexus-indexer/src/main/java/org/sonatype/nexus/index/DefaultQueryCreator.java
 
        // A WildcardQuery supports the * and ? wildcard characters
        Query q = new WildcardQuery( new Term( SampleIndexCreator.MY_FIELD, "*alue" ) );
 
        Set artifacts = app.searchIndexFlat( q );
 
        assertNotNull( "returned artifacts is null", artifacts );
        assertFalse( "returned artifacts is empty", artifacts.isEmpty() );
 
        for ( ArtifactInfo ai : artifacts )
        {
            assertEquals( "returned artifact has invalid data", "value", ai.getAttributes().get( SampleIndexCreator.MY_FIELD ) );
        }
 
        // A WildcardQuery supports the * and ? wildcard characters
        q = new WildcardQuery( new Term( SampleIndexCreator.MY_FIELD, "v?lue" ) );
 
        artifacts = app.searchIndexFlat( q );
 
        assertNotNull( "returned artifacts is null", artifacts );
        assertFalse( "returned artifacts is empty", artifacts.isEmpty() );
 
        for ( ArtifactInfo ai : artifacts )
        {
            assertEquals( "returned artifact has invalid data", "value", ai.getAttributes().get( SampleIndexCreator.MY_FIELD ) );
        }
 
        // A WildcardQuery supports the * and ? wildcard characters
        q = new WildcardQuery( new Term( SampleIndexCreator.MY_FIELD, "val*" ) );
 
        artifacts = app.searchIndexFlat( q );
 
        assertNotNull( "returned artifacts is null", artifacts );
        assertFalse( "returned artifacts is empty", artifacts.isEmpty() );
 
        for ( ArtifactInfo ai : artifacts )
        {
            assertEquals( "returned artifact has invalid data", "value", ai.getAttributes().get( SampleIndexCreator.MY_FIELD ) );
        }
    }
 
    public void testSampleSearchGroup()
        throws Exception
    {
        app.index();
 
        Map groupedArtifacts = app.searchIndexGrouped( SampleIndexCreator.MY_FIELD, "value" );
 
        assertNotNull( "returned groupedArtifacts is null", groupedArtifacts );
        assertFalse( "returned groupedArtifacts should not be empty", groupedArtifacts.isEmpty() );
 
        for ( ArtifactInfoGroup artifactGroup : groupedArtifacts.values() )
        {
            String[] parts = artifactGroup.getGroupKey().split( ":" );
            //1st part groupId
            //2nd part artifactId
            //3rd part version
            //4th part classifier
            assertEquals( "should be 4 parts to the group key", 4, parts.length );
            assertFalse( "each group should contain at least 1 artifact", artifactGroup.getArtifactInfos().isEmpty() );
        }
    }
 
    public void testSampleSearchGroupNewGrouping()
        throws Exception
    {
        app.index();
 
        // Search using my own grouping, which will group based upon the MY_FIELD parameter
        Map groupedArtifacts = app.searchIndexGrouped(
            SampleIndexCreator.MY_FIELD,
            "value",
            new AbstractGrouping()
            {
                @Override
                protected String getGroupKey( ArtifactInfo artifactInfo )
                {
                    return artifactInfo.getAttributes().get( SampleIndexCreator.MY_FIELD );
                }
            } );
 
        assertNotNull( "returned groupedArtifacts is null", groupedArtifacts );
        assertEquals( "returned groupedArtifacts should have 1 entry", 1, groupedArtifacts.size() );
        assertEquals( "group key should be value", "value", groupedArtifacts.values().iterator().next().getGroupKey() );
    }
}

As you can see from the last test method, you can create your own Grouping object, to group things anyway you see fit, in this case, the results will be grouped using the MY_FIELD parameter.

That’s all for now, next time we will get into packing up indexes for publishing.

Sample Maven project can be found here http://svn.sonatype.org/nexus/trunk/sandbox/nexus-indexer-sample and will be updated periodically as I put together more details for the blog posts.

Nexus Indexer API: Part 1

Damian Bradicich — Mon, 01 Jun 2009 10:00:46 +0000

Indexing
Searching
Packing
Updating

This post will focus on Indexing and provide you with some concrete code examples that show you how to integrate the Nexus Indexer into your own application.

Indexing

The Nexus Indexer is responsible for looking at a maven repository directory layout, and generating a searchable index of the contents of that repository. Below I have a very simple Plexus component that uses the Nexus Indexer to index a directory:

package org.damian;
 
import java.io.File;
import java.io.IOException;
import java.util.List;
 
import org.codehaus.plexus.component.annotations.Component;
import org.codehaus.plexus.component.annotations.Configuration;
import org.codehaus.plexus.component.annotations.Requirement;
import org.codehaus.plexus.personality.plexus.lifecycle.phase.Disposable;
import org.codehaus.plexus.personality.plexus.lifecycle.phase.Initializable;
import org.codehaus.plexus.personality.plexus.lifecycle.phase.InitializationException;
import org.sonatype.nexus.index.NexusIndexer;
import org.sonatype.nexus.index.context.IndexCreator;
import org.sonatype.nexus.index.context.IndexingContext;
import org.sonatype.nexus.index.context.UnsupportedExistingLuceneIndexException;
 
/**
 * Sample app to show how to integrate with the nexus indexer.  Note that this is a simple plexus
 * component extending the SampleApp interface
 *
 * public interface SampleApp
 * {
 *    void index()
 *        throws IOException;
 * }
 *
 * @author Damian
 *
 */
@Component( role = SampleApp.class )
public class DefaultSampleApp
    implements SampleApp,
        Initializable,
        Disposable
{
    // The nexus indexer
    @Requirement
    private NexusIndexer indexer;
 
    // The list of index creators we will be using (all of them)
    @Requirement( role = IndexCreator.class )
    private List indexCreators;
 
    // The indexing context
    private IndexingContext context = null;
 
    // The path to the repository to index, value will be pulled from
    // the plexus context
    @Configuration( value = "${repository.path}" )
    private File repositoryDirectoryPath;
 
    // The path to store index files, value will be pulled from
    // the plexus context
    @Configuration( value = "${index.path}")
    private File indexDirectoryPath;
 
    // Initialize the index context
    public void initialize()
        throws InitializationException
    {
        try
        {
            // Add the indexing context
            context = indexer.addIndexingContext(
                // id of the context
                "sample",
                // id of the repository
                "sampleRepo",
                // directory containing repository
                repositoryDirectoryPath,
                // directory where index will be stored
                indexDirectoryPath,
                // remote repository url...not in this example
                null,
                // index update url...not in this example
                null,
                // list of index creators
                indexCreators );
        }
        catch ( UnsupportedExistingLuceneIndexException e )
        {
            throw new InitializationException( "Error initializing IndexingContext", e );
        }
        catch ( IOException e )
        {
            throw new InitializationException( "Error initializing IndexingContext", e );
        }
    }
 
    // clean up the context
    public void dispose()
    {
        if ( context != null )
        {
            // Remove the index files, typically would not want to remove the index files, so
            // would pass in false, but this is just a test app...
            try
            {
                indexer.removeIndexingContext( context, true );
            }
            catch ( IOException e )
            {
                e.printStackTrace();
            }
        }
    }
 
    // index the repository
    public void index()
        throws IOException
    {
        // Perform the scan, which will index all artifacts in the repository directory
        // once this is done, searching will be available
        indexer.scan( context );
    }
}

As you can see, there isn’t much code required to properly manage indexes of your Maven 2 repository. This is the simplest case, where you don’t have anything you want to add to the indexer, but suppose you want to index additional data.

The Nexus Indexer uses a set of IndexCreator Plexus components to create the Lucene indexes that store the data. The Nexus Indexer has 2 by default, the MinimalArtifactInfoIndexCreator (min) and the JarFileContentsIndexCreator (jar). The min creator takes common items from an artifact and pom (will do best guess if a pom isn’t available) such as groupId, artifactId, version, packaging, classifier and puts those in the index as searchable fields. The jar creator simply puts all class names into the index as searchable fields. Of course you can create your own IndexCreator Plexus component that can index whatever you see fit.

I will now show a sample creator that will explain how to add a new IndexCreator (Note that nexus-indexer-2.0.1-SNAPSHOT is required for this sample to run properly)

package org.sonatype.nexus.index.creator;
 
import java.io.IOException;
 
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.codehaus.plexus.component.annotations.Component;
import org.sonatype.nexus.index.ArtifactContext;
import org.sonatype.nexus.index.ArtifactInfo;
import org.sonatype.nexus.index.context.IndexCreator;
 
/**
 * A Sample Index Creator that will show how to properly create your own IndexCreator component
 * @author Damian
 *
 */
// Define the plexus component, and the hint that plexus will use to load it
@Component( role = IndexCreator.class, hint = "sample" )
public class SampleIndexCreator
    extends AbstractIndexCreator
{
    // The name of my sample field
    public static final String MY_FIELD = "myfield";
 
    /**
     * Populate ArtifactInfo with data specific to your application.  Note that the
     * artifactContext contains other useful objects, which may come in handy.
     */
    public void populateArtifactInfo( ArtifactContext artifactContext )
        throws IOException
    {
        // Add the data to the ArtifactInfo
        artifactContext.getArtifactInfo().addAttribute( MY_FIELD, "value" );
    }
 
    /**
     * Popluate ArtifactInfo from exisiting lucene index document, will want to populate the
     * same fields that you populate in populateArtifactInfo
     */
    public boolean updateArtifactInfo( Document document, ArtifactInfo artifactInfo )
    {
        // Add the data to the ArtifactInfo
        artifactInfo.addAttribute( MY_FIELD, document.get( MY_FIELD ) );
 
        //Note that returning false here will notify calling party of failure
        return true;
    }
 
    /**
     * Add data from the artifactInfo to the index
     */
    public void updateDocument( ArtifactInfo artifactInfo, Document document )
    {
        document.add(
            new Field(
                //Field name, should be unique across all IndexCreator objects
                MY_FIELD,
                // get your new data and add to the index
                artifactInfo.getAttribute( MY_FIELD ),
                // Whether the data should be stored (YES or NO)
                Field.Store.YES,
                // Whether the field should be indexed (NO, TOKENIZED, UNTOKENIZED)
                Field.Index.UN_TOKENIZED ) );
    }
}

Simply pass this creator in with the list of creators when adding an index context, and it will now be available. And that’s it for this entry, next time we’ll get to searching the nexus index.

Sample Maven project can be found here http://svn.sonatype.org/nexus/trunk/sandbox/nexus-indexer-sample and will be updated periodically as I put together more details for the blog posts.