Variance in Scala (“Luke, he is your father too”)

When working with Big Data, sometimes it’s useful to remember that powerful products wouldn’t work properly without the tools that build them. It’s possible to start programming in Scala with a few case classes and a bunch of for-comprehensions, but those are only little scratches in a huge ice surface like Scala is. It may not be enough to make your code clean and comprehensible.  I’ve been developing with this programming language for almost 4 years, and every day I discover a new feature that surprises me. That acknowledgement, in the end, is the main reason to keep digging deeper into Scala.

Stratio’s Lucene-based index for Cassandra is now a plugin

Thanks to the changes proposed at CASSANDRA-8717CASSANDRA-7575 and CASSANDRA-6480, Stratio is glad to present its Lucene-based implementation of Cassandra secondary indexes as a plugin that can be attached to the Apache distribution. Before the above changes, Lucene index was distributed inside a fork of Apache Cassandra, with all the difficulties it implied, i.e. maintaining a fork. As of now, the fork is discontinued and new users should use the recently created plugin, which maintains all the features of Stratio Cassandra.

 

Stratio’s Lucene index extends Cassandra’s functionality to provide near real-time distributed search engine capabilities such as with ElasticSearch or Solr, including full text search capabilities, free multivariable search,

MongoDB – Spark Connector Whitepaper

We recently worked with MongoDB and their developer team for the analysis of their Hadoop based connector Vs our native connector solution. The paper highlights how Stratio’s connector for Apache Spark implements the PrunedFilteredScan API instead of the TableScan API which effectively allows you to avoid scanning the entire collection.

Our connector supports the Spark Catalyst optimizer for both rule-based and cost-based query optimization.

Stratio Release 1.2.0

  • Added HDFS as an option of persistence technologies.
  • Added HDFS Connector for Crossdata.
  • Retry button when a node fails to install.
  • UX refactor.
  • Now the Admin machine is the package repository for node installation (no 3rd party repositories needed).
  • Backend bugfixes.
  • Uninstall scripts for RedHat/CentOs.

Top-k queries in Cassandra: An embedded mapreduce approach

Stratio has just added top-k queries support to its Lucene based implementation of the Cassandra’s secondary indexes. This implementation was originally designed to allow embedded full-text and multivariable search in Apache Cassandra. The previous release included an ad-hoc mechanism to perform distributed relevance queries based on the Lucene’s scoring algorithm. The current release generalizes this mechanism to allow several types of top-k queries.

Spark-MongoDB library

Once Data Sources API  has been released, we’ve wanted to take advantage of these new features and, for this reason, we have developed a Spark-MongoDB library. With this new connector we help the growing MongoDB community to simplify the interaction with this datasource via Spark.

This library provides the mechanism for accessing MongoDB collections in a structured way from SparkSQL, accesible from Python and Scala API’s. Since MongoDB is an open-source document database leader among NoSQL databases and is highly used in several projects [http://www.mongodb.com/leading-nosql-database] we find this connection with all the operations permitted by SparkSQL not only useful but necessary.

When Stratio Met Spark: A True Love Story

Certified distribution

Stratio is delighted to announce that it is officially a Certified Spark Distribution. The certification is very important for us because we deeply believe that the certification program provides many benefits to the Spark community: It facilitates collaboration and integration, offers broad evolution and support for the rich Spark ecosystem, simplifies adoption of critical security updates and allows development of applications valid for any certified distribution – a key ingredient for a successful ecosystem.

Paper of the week: “BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data” [1]

This paper has been presented at the Eurosys 2013 conference and is avaiblable for download at the conference website. The paper presents BlinkDB that, despite its name, is not a database but a query engine on top of Hive and Shark, and it is used for running interactive SQL queries on large volumes of data using data samples. BlinkDB is built using two key ideas: an adaptive optimization framework to build and maintain stratified samples, and a dynamic sample selection strategy to select appropiately sized sample based on a query’s accuracy or response time requirements.
This paper offers an interesting introduction on how to apply statistical inference technics on Big Data and makes clear that there is always a trade-off between accuracy and performance. In that regard, BlinkDB offers information about query accuracy so the user can make decisions. Although it is not clear what the cost of maintaining stratified samples is, the paper provides a good seed for future works in the area.
[1] Agarwal, Sameer, et al. “BlinkDB: queries with bounded errors and bounded response times on very large data.” Proceedings of the 8th ACM European Conference on Computer Systems. ACM, 2013.