With the release of Spark version 2.0, streaming starts becoming much more accessible to users. By adopting a continuous processing model (on an infinite table), the developers of Spark have enabled users of its SQL or DataFrame APIs to extend their analytic capabilities to unbounded streams.

Within the Spark community, Databricks Engineer, Michael Armbrust is well-known for having led the long-term project to move Spark’s interactive analytics engine from Shark to Spark SQL. (Full disclosure: I’m an advisor to Databricks.) Most recently he has turned his efforts to helping introduce a much simpler stream processing model to Spark Streaming (“structured streaming”).

Structured Streaming Comes to Apache Spark 2.0

O'Reilly Media spreads the knowledge of innovators. At O’Reilly, a big part of our business is paying attention to what’s new and interesting in the world of technology. We have a pretty good record at having anticipated some of the big technology developments in recent history. For instance, we launched the first commercial Web site, GNN, in 1993; we organized the meeting at which the term “open source” was first adopted; we were early investors in Blogger, which helped launch the blogging revolution; and more recently, our Web 2.0 conference launched a world-wide meme. We call this predictive sense the “O’Reilly Radar.” And while we’re certainly not always right, we are, at least, good at making interesting guesses.

Our methodology is simple: we draw from the wisdom of the alpha geeks in our midst, paying attention to what’s interesting to them, amplifying these weak signals, and seeing where they fit into the innovation ecology. Add to that the original research conducted by our Research team, and you start to get a good picture of what the technology world is thinking about.

Structured Streaming Comes to Apache Spark 2.0

O'Reilly Radar