Stratio’s Lucene-based index for Cassandra is now a plugin

Thanks to the changes proposed at CASSANDRA-8717CASSANDRA-7575 and CASSANDRA-6480, Stratio is glad to present its Lucene-based implementation of Cassandra secondary indexes as a plugin that can be attached to the Apache distribution. Before the above changes, Lucene index was distributed inside a fork of Apache Cassandra, with all the difficulties it implied, i.e. maintaining a fork. As of now, the fork is discontinued and new users should use the recently created plugin, which maintains all the features of Stratio Cassandra.

 

Stratio’s Lucene index extends Cassandra’s functionality to provide near real-time distributed search engine capabilities such as with ElasticSearch or Solr, including full text search capabilities, free multivariable search, relevance queries and field-based sorting. Each node indexes its own data, so high availability and scalability is guaranteed.

 

With the new distribution as a plugin you can easily patch an existing installation of Cassandra with the following command:

 mvn clean package -Ppatch -Dcassandra_home=<CASSANDRA_INSTALL_DIR>

Now, given an existing table with the following schema


 

you can create a Lucene index with this query:


 

Once the index has been created, we can start issuing queries. For instance, retrieve 10 tweets containing the word Cassandra in the message body:


 

We can also find the 5 best matching tweets containing the word Cassandra in the message body for the previous query, involving Lucene relevance features:


 

Queries, filters and boolean predicates can be combined to do queries as complex as requesting the tweets inside a certain time range starting with an ‘A’ or containing a ‘B’, or with the word ‘FAST’, sorting them by ascending time, and then by descending alphabetic user name:


 

You can learn more about Stratio’s Lucene index for Cassandra reading the comprehensive documentation or viewing the video of its presentation at Cassandra Summit Europe 2014.

We will be back very soon with new exciting features such as geospatial search and bitemporal indexing.

 

6 Comments

  1. I am using the plugin on version 3.0.9. I am trying to do a very similar thing to the complex query you posted here, but keep getting the following: Error from server: code=2200 [Invalid query] message=”Unparseable JSON search: {…}”.

    So I copied these queries from your blog and tried them out on my server – same thing. Any ideas? How complex can these nested boolean queries be? The query verbatim:

    SELECT * FROM tweets WHERE lucene = ‘{
    filter :
    {
    type : “boolean”, must :
    [
    {type : “range”, field : “time” lower : “2014/04/25”},
    {type : “boolean”, should :
    [
    {type : “prefix”, field : “user”, value : “a”} ,
    {type : “wildcard”, field : “user”, value : “*b*”} ,
    {type : “match”, field : “user”, value : “fast”}
    ]
    }
    ]
    },
    sort :
    {
    fields: [ {field :”time”, reverse : true},
    {field : “user”, reverse : false} ]
    }
    }’ LIMIT 10000;

    The response:

    InvalidRequest: Error from server: code=2200 [Invalid query] message=”Unparseable JSON search: {
    filter :
    {
    type : “boolean”, must :
    [
    {type : “range”, field : “time” lower : “2014/04/25”},
    {type : “boolean”, should :
    [
    {type : “prefix”, field : “user”, value : “a”} ,
    {type : “wildcard”, field : “user”, value : “*b*”} ,
    {type : “match”, field : “user”, value : “fast”}
    ]
    }
    ]
    },
    sort :
    {
    fields: [ {field :”time”, reverse : true},
    {field : “user”, reverse : false} ]
    }
    }”

    • Thanks for your interest in the project.

      It seems that there are several mistakes in the example query. Specifically, there is a missing comma in the 6th line, right after “time”, and the field names are wrong. The proper query would be:

      SELECT * FROM tweets WHERE lucene = ‘{
      filter :
      {
      type : “boolean”, must :
      [
      {type : “range”, field : “createdat”, lower : “2014-04-25”},
      {type : “boolean”, should :
      [
      {type : “prefix”, field : “userid”, value : “a”} ,
      {type : “wildcard”, field : “userid”, value : “*b*”} ,
      {type : “match”, field : “userid”, value : “fast”}
      ]
      }
      ]
      },
      sort :
      {
      fields: [ {field :”createdat”, reverse : true},
      {field : “userid”, reverse : false} ]
      }
      }’ LIMIT 10000;

      Also, please note that you should use vertical quotation marks and apostrophes (” and ‘) instead of their curly counterparts (“,” and ’).

      We are fixing the example, thanks for reporting.

  2. Hi,

    I believe the query be:

    SELECT * FROM tweets WHERE lucene = ‘{
    filter : {type : “match”,
    field : “message”,
    value : “cassandra”}
    }’ LIMIT 10;

    with “message” instead of “text”.

    • Eduardo Alonsosays:

      Thank you Valeriu, you are totally right. We fixed it

  3. Hi,
    first of all thumbs up for great work done with lucene index !!!
    Now for my question:

    can you share in few words with us the internals of the implementation…
    I see that we can define some refresh time in the index options.
    As I understand the index itself is maintained by C* in a synchronous manner, the refresh time is more lucene internal.
    Is there a kind of index caching ? Is the refresh defining the cache invalidation/refresh time?
    Also can you share some details about resource consumption? Is the cache in memory, what about space requirements and limits?

    Also a question that came up… how do I know that the refresh is working as defined? Could in theory the refresh stop to do hos work and my queries work on stale data?

    many thanks in advance for your kind feedback
    cheers
    Heiko

Leave a comment

Please be polite. We appreciate that. Your email address will not be published and required fields are marked