In this episode of the Data Show, I spoke with Jason Dai, CTO of big data technologies at Intel, and co-chair of Strata + Hadoop World Beijing. Dai and his team are prolific and longstanding contributors to the Apache Spark project. Their early contributions to Spark tended to be on the systems side and included Netty-based shuffle, a fair-scheduler, and the “yarn-client” mode. Recently, they have been contributing tools for advanced analytics. In partnership with major cloud providers in China, they’ve written implementations of algorithmic building blocks and machine learning models that let Apache Spark users scale to extremely high-dimensional models and large data sets. They achieve scalability by taking advantage of things like data sparsity and Intel’s MKL software. Along the way, they’ve gained valuable experience and insight into how companies deploy machine learning models in real-world applications.

Deep Learning for Apache Spark

O'Reilly Media spreads the knowledge of innovators. At O’Reilly, a big part of our business is paying attention to what’s new and interesting in the world of technology. We have a pretty good record at having anticipated some of the big technology developments in recent history. For instance, we launched the first commercial Web site, GNN, in 1993; we organized the meeting at which the term “open source” was first adopted; we were early investors in Blogger, which helped launch the blogging revolution; and more recently, our Web 2.0 conference launched a world-wide meme. We call this predictive sense the “O’Reilly Radar.” And while we’re certainly not always right, we are, at least, good at making interesting guesses.

Our methodology is simple: we draw from the wisdom of the alpha geeks in our midst, paying attention to what’s interesting to them, amplifying these weak signals, and seeing where they fit into the innovation ecology. Add to that the original research conducted by our Research team, and you start to get a good picture of what the technology world is thinking about.

Deep Learning for Apache Spark

O'Reilly Radar