The Developer’s Guide to Scala Implicit Values (Part II)

Imagine a rectangular grid of cells, in which each cell has a value – Either black (dead) or white (alive). And imagine that:

  1. Any live cell with two or three live neighbors survives for the next generation.
  2. Any cell with four or more neighbors dies from overpopulation.
  3. Any cell with one or no neighbors dies from isolation.
  4. Any dead cell with exactly three neighbors comes to life.

 

These are the four simple rules of Conway’s Game of Life . You could hardly imagine a simpler set of rules to code on your computer and you wouldn’t expect any interesting result at all, but…

Behold the wonders of its hidden might!

Continuous delivery in depth #1

Following on from a previous “Lunch & learn” about how Jenkins is being used for Stratio’s Continuous Delivery jobs (watch on Stratio’s youtube channel), it seemed logical to provide more information on our Jenkins pipeline plug-in usage.

In this first issue, we will  follow how pipelines are being used at Stratio Big Data to achieve full lifecycle traceability, from the development team to a final productive environment.

Some pitfalls were mentioned during the “Lunch & Learn” meeting and will be explained to help you fully comprehend the nature of the underlying bug and the solution achieved. This will follow in a second issue.

How to Aggregate Data in Real-Time with Stratio Sparta

When working with Big Data, it’s frequent to have the need to aggregate data in real-time, whether it comes from a specific service, such as social networks (Twitter, Facebook…) or even from more diverse sources, like a weather station. A good way to process these large amounts of information is with Spark Streaming, it provides us all the data in real time, but it has one problem: you have to program it yourself.

Variance in Scala (“Luke, he is your father too”)

When working with Big Data, sometimes it’s useful to remember that powerful products wouldn’t work properly without the tools that build them. It’s possible to start programming in Scala with a few case classes and a bunch of for-comprehensions, but those are only little scratches in a huge ice surface like Scala is. It may not be enough to make your code clean and comprehensible.  I’ve been developing with this programming language for almost 4 years, and every day I discover a new feature that surprises me. That acknowledgement, in the end, is the main reason to keep digging deeper into Scala.

Stratio’s Lucene-based index for Cassandra is now a plugin

Thanks to the changes proposed at CASSANDRA-8717CASSANDRA-7575 and CASSANDRA-6480, Stratio is glad to present its Lucene-based implementation of Cassandra secondary indexes as a plugin that can be attached to the Apache distribution. Before the above changes, Lucene index was distributed inside a fork of Apache Cassandra, with all the difficulties it implied, i.e. maintaining a fork. As of now, the fork is discontinued and new users should use the recently created plugin, which maintains all the features of Stratio Cassandra.

 

Stratio’s Lucene index extends Cassandra’s functionality to provide near real-time distributed search engine capabilities such as with ElasticSearch or Solr, including full text search capabilities, free multivariable search,

MongoDB – Spark Connector Whitepaper

We recently worked with MongoDB and their developer team for the analysis of their Hadoop based connector Vs our native connector solution. The paper highlights how Stratio’s connector for Apache Spark implements the PrunedFilteredScan API instead of the TableScan API which effectively allows you to avoid scanning the entire collection.

Our connector supports the Spark Catalyst optimizer for both rule-based and cost-based query optimization.

Stratio Release 1.2.0

  • Added HDFS as an option of persistence technologies.
  • Added HDFS Connector for Crossdata.
  • Retry button when a node fails to install.
  • UX refactor.
  • Now the Admin machine is the package repository for node installation (no 3rd party repositories needed).
  • Backend bugfixes.
  • Uninstall scripts for RedHat/CentOs.

Top-k queries in Cassandra: An embedded mapreduce approach

Stratio has just added top-k queries support to its Lucene based implementation of the Cassandra’s secondary indexes. This implementation was originally designed to allow embedded full-text and multivariable search in Apache Cassandra. The previous release included an ad-hoc mechanism to perform distributed relevance queries based on the Lucene’s scoring algorithm. The current release generalizes this mechanism to allow several types of top-k queries.

Spark-MongoDB library

Once Data Sources API  has been released, we’ve wanted to take advantage of these new features and, for this reason, we have developed a Spark-MongoDB library. With this new connector we help the growing MongoDB community to simplify the interaction with this datasource via Spark.

This library provides the mechanism for accessing MongoDB collections in a structured way from SparkSQL, accesible from Python and Scala API’s. Since MongoDB is an open-source document database leader among NoSQL databases and is highly used in several projects [http://www.mongodb.com/leading-nosql-database] we find this connection with all the operations permitted by SparkSQL not only useful but necessary.