When you hear about a conference rebranded as Big Things, several different scenarios likely cross your mind. Just about everything fits in the Big Things box… However, once you step foot into the conference, with the largest cinema in the world playing host, you immediately know what it is all about.
Big Data Spain is one of the three largest conferences of its kind in Europe. The conference focuses primarily on Big Data, Artificial Intelligence, Cloud Technologies and Digital Transformation. #BDS17 will take place this very 15, 16 and 17 of November right here in sunny Madrid. So, don´t miss out because there is going to be a treasure trove of prolific speakers such as Paco Nathan, Jim Webber and Phillip Krenn.
It all started with a call from Adrian Arias, Enterprise Relationship Manager at LinkedIn, a morning at the beginning of September: “Congratulations Stratio, you are one of our finalists to the “Premios IN”, nominated in the Best Recruitment Team category. Only a 5% of our clients make it to the nomination”.
Our participation at Spark Summit 2017 in San Francisco was once again a great experience. Not only because of the quality of the speakers, but the special atmosphere built up by Spark lovers. In order to enliven the breaks between sessions talks, we brought a riddle, inviting all attendees to solve it.
On 12-13 May 2017, the 6th edition of UX Spain took place. If there was a scale enabling us to measure when an event has reached its maturity, UX Spain would be, after this edition, at the top. Once again, an impeccable organization, outstanding speakers with juicy content and two consecutive days as a trending topic on Twitter have turned UXSpain into a reference in the Design and UX sector. Well done to the organizers!
By Sondos Atwi @Sondos_4
On the 17th and 18th of November, I attended the Big Data Spain conference. It was my first time attending this type of events, and it was an excellent opportunity to meet experts in the fields and attend high-quality talks. So I decided to write this post to share a few of the presented slides and ideas.
Ps: Please excuse the quality of some slides/pictures, they were all taken by my phone camera!
First, Congrats to Big Data Spain on being the second biggest Big Data conference in Europe, right after O’Reilly Strata. This year’s edition also
had around 50% increase than last year’s!
Now let’s dig into the details…
MesosCon Europe was held in Amsterdam from August 31 to September 2 and a small representation of Stratio’s crew was over there.
Benjamin Hindman’s opening keynote
Mesosphere’s Co-Founder & Chief Architect Benjamin Hindman broke the ice with the first keynote.
After talking about the mesos ecosystem’s growth within the last months, he explained the nested containerization model and the improvements in the Mesos resource allocation. He then introduced an African nonprofit organization called praekelt.org dedicated to using mobile technology to improve the lives of people living in poverty. A representative from the NGO explained how mesos and DC/OS are a perfect fit for its cluster provisioning. The NGO has to run quite a few clusters where 80% of the setup is identical for each cluster and the remaining 20% is different. They therefore get the most out of mesos and DC/OS by deploying these distinctive features separately. The representative was asked what the Mesos’ biggest deficiency is nowadays. He replied that they were struggling to find a persistence layer that fits their current needs (he pointed out that they are currently using GlusterFS as their persistence backend).
Netflix’s Sharma Podila on stage at Mesos
Senior Software Engineer Sharma Podila at Netflix closed the first round of keynotes. He started by talking about Fenzo, a scheduler java library for Mesos that contains some plugins and strategies that might be used to optimize common tasks when creating a Mesos Framework. At Stratio we were playing around with this library when we created our very own framework for Stratio Decision.
Podila also mentioned Project Titus which is Netflix’s internal container cloud that runs both batch-style jobs and services. A question was raised as to whether Netflix is planning on open-sourcing Titus, but Podila explained that the platform is very coupled to the Netflix architecture.
After the keynotes, we attended the “Mesos 1.0” talk delivered by Vinod Kone and Anand Mazumbar who both work for Mesosphere. They explained the new release cycle defined for Mesos which aims at releasing a new version every 2 months. Every minor release will be supported for a period of 6 months. More information here: mesos versioning. Their talk detailed the efforts they have been making over the past few months to improve the Mesos APIs and provided information on their brand new (still experimental) Operator API.
The evolution of Deploy Tooling at Twitter
Another interesting speech – “The evolution of Deploy Tooling at Twitter” – was delivered by David McLaughlin. When deploying microservices at twitter, the team met with challenges and made some mistakes.
He pointed out that when you deploy something in production, it is always best to have a simple and fast mechanism to rollback when things do not go as expected.
Regarding Apache Aurora, McLaughlin said that it could be seen as Twitter’s Marathon, although it also supports scheduling batch tasks or cron jobs (as Chronos does). He explained how Aurora is helping Twittter with its Canary deployment which is a production deployment strategy that allows rolling out updates to a subset of job instances in order to test different code versions alongside the actual production job. See the following link for more info: Martin Fowler
McLaughlin also highlighted that Aurora supports preemption in order to guarantee that important production jobs are always running. Check out the following link for further info: Aurora preemption.
The speech ended with a demo of Apache Workflows – Deploying on Apache Aurora at Twitter with the following design goals: introducing a new entity – a Workflow – that captures all testing and release steps, supports and encourages continuous deployment and makes rollbacks a first class citizen of the tool.
Marathon and Chronos
Matthias Eichstedt and Matthias Veit from Mesosphere talked about Marathon and Chronos State of the Art. Marathon ensures long-running tasks are running in the datacenter and is commonly used to maintain high availability of other frameworks. Chronos is the fault tolerant cron of the datacenter. The roadmap for each of these projects revealed that Chronos has been replaced by Metronome, a brand-new Mesos framework for scheduled jobs. If you are interested in this project check out its github: Metronome github. Metronome has been integrated natively with the brand new DC/OS 1.8 and is available on the Jobs tab on the DC/OS UI.
During the lunch break we had a friendly and interesting chat with the people from REX-Ray about our current persistence needs at Stratio.
After lunch, Anindya Sinha from Apple Inc. explained a cool new feature of Mesos that will be available in future versions: supporting the sharing of resources across task instances scheduled on a single agent node. At the moment, a persistent volume, once offered to the framework, can be scheduled to a task, but cannot be used by another task until the former has finished. This new feature will add support to enable shareability for persistent volumes. See the following jira ticket for further details MESOS-3421
Building highly available Mesos frameworks
The first day’s last talk was a very interesting one. Neil Conway from Mesosphere talked about Building highly available Mesos frameworks. His talk offered practical information to understand how Mesos deals with failure and the tools it provides to enable fault tolerant frameworks. Throughout his slides he talked about three types of failures: network, process and system failures and how mesos and the frameworks developers should work to deal with them. There is also a pretty good Mesos documentation on this same topic: Designing highly available Mesos frameworks
To conclude his talk, Neil talked about some changes that will be included in Mesos 1.1: operators/scripts will be able to tell Mesos that a partitioned agent is permanently gone, frameworks will be able to learn when a partitioned task is gone “forever” and a fine-grained task states, replacing TASK_LOST (TASK_UNREACHABLE, TASK_DROPPED, TASK_GONE & TASK_GONE_BY_OPERATOR).
Work hard, play hard
After attending a day of interesting and insightful talks, we were able to enjoy a stroll along the famous Amsterdam canals.
Kris Buytaert’s opening keynote
We arrived bright and early for the second day of the event. Kris Buytaert, CTO of Inuits delivered his keynote on the impact of distributed computing on the evolution of Devops a movement that started with the devopsday held in Gent in November 2009. Buytaert explained how the distributed platforms have deeply impacted the way we deliver software.
Ambrose: automatic metric collection and correlation in DC/OS
We then decided to listen to Mesosphere’s Nick Parker talking about “Ambrose: automatic metric collection and correlation in DC/OS”. Parker highlighted the importance of using metrics on a daily basis and how they may help foresee problems. He then explained the architecture of Ambrose, a suite of components for collecting and tagging metrics in the DC/OS system:
The points to be highlighted are that metrics are automatically retrieved from the containers, custom applications might publish their metrics using the StatsD endpoint, so far they are flushing the metrics to a kafka cluster (more outputs to come in the future), they are using Apache Avro for serialization and Grafana for visualization.
Jan Schlicht’s talk on “Docker at Scale” closed our MesosCon Europe experience off nicely.
Schlicht focused on why everybody is talking about containers and how they make the lives of devs and ops easier. He then explained the differences between docker and mesos containerizer and things that may go wrong when running containers at scale.
These include: hanging docker daemons, performance drawbacks because of having too many docker layers and exhausting the agent inodes.
We most definitely had an intense and exciting time at MesosCon Europe. The two-day speech-packed event confirmed that the Mesos ecosystem is packing a punch.
Friday, July 31st is the SysAdmin Appreciation Day and at Stratio we really want to go all out and celebrate it. During the entire day we will be giving sysadmin related talks and workshops.
Here is the list of what you’ll be able to enjoy:
The Cassandra Summit 2014 took place on September 10-12 in San Francisco with more than 2000 attendants. The summit was split into 3 days, with Training sessions on the first and third day and Conference sessions with 6 tracks on the second one. The first thing we noticed after passing through the registration desk was how amazing the organization was.