Favorites

Jonas Bonér, CTO of LightBend and creator Akka, discusses using Akka when developing distributed systems. He talks about the Actor Model, and how every Microservice needs to be viewed as a system to be successful. 

Why listen to this podcast:

 - Akka is JVM-based framework design for developing distributed systems leveraging the Actor Model - an approach for writing concurrent systems that treat actors as universal primitives and the most successful model with abstraction has been streaming
 - Circuit breakers in Akka are a backup and retry policy; they protect you by capturing failure data and allow you to roll back 
 - Every Microservice needs to be viewed as a system, it needs to have multiple parts that run on different machines in order to function and be fully resilient - thus is a Microsystem
 - Two different trends have emerged when it comes to hardware and environments: one is the trend toward Multi-core, the is a movement toward virtualized environments and the cloud
 - Saga pattern of managing long running transactions in a distributed system fits very well with messaging style architectures

Notes and links can be found on: http://bit.ly/2kwB2eB

Akka
The Actor Model
When Akka and the Actor Model is the perfect choice
Circuit breakers patterns in distributed systems
Two trends toward Multi-core
Reactive Manifesto
Event Driven vs. Message Driven
Reactive Programming and Streams
Microliths to Microsystems
What do you have to get right before you start trying to deploy a distributed systems?
Working with ML / AI at Lightbend to understand tracing through distributed system
Saga Pattern

More on this:
Quick scan our curated show notes on InfoQ http://bit.ly/2kwB2eB

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

InfoQ Podcast

Show Notes

Jonas Bonér on the Actor Model, Akka, Reactive Programming, Microservices and Distributed Systems

On this week’s podcast, Danny Yuan, Uber’s Real-time Streaming/Forecasting Lead, lays out a thorough recipe book for building a real-time streaming platform with a major focus on forecasting. In this podcast, Danny discusses everything from the scale Uber operates at to what the major steps for training/deploy models in an iterative (almost Darwinistic) fashion and wraps with his advice for software engineers who want to begin applying machine learning into their day-to-day job.  

Why listen to this podcast:

* Uber processes 850,000 - 1.3 million messages per second in their streaming platform with about 12 TB of growth per day. The system’s queries scan 100 million to 4 billion documents per second.
* Uber’s frontend is mobile. The frontend talks to an API layer. All services generate events that are shuffled into Kafka. The real-time forecasting pipeline taps into Kafka to processes events and stores the data into Elasticsearch. * There is a federated query layer in front of Elasticsearch to provide OLAP query capabilities.
* Apache Flink’s advanced windowing features, programming model, and checkpointing convinced Uber to move away from the simplicity of Apache Samza.
* The forecasting system allows Uber to remove the notion of delay by using recent signals plus historical data to project what is happening now and what will happen into the future.
* Uber’s pipeline for deploying ML models: HDFS, feature engineering, organizing into data structures (similar to data frames), deploy mostly offline training models, train models, & store into a container-based model manager. 
* A model serving layer is used to pick which model to use, forecasting results are stored in an OLAP data store, a validation layer compares real results against forecast results to verify the model is working as desired, and a rollback feature enables poor performing models to be automatically replaced by previous one.
* “Without output, you don’t have input.” If you want to start leveraging machine learning, developers just need to start doing. Start with intuition and practice. Over time ask questions and learn what you need, then apply a laser focus to gain that knowledge.

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Check the landing page on InfoQ: https://bit.ly/2GJQbUo

Streaming: Danny Yuan on Real-Time, Time Series Forecasting @Uber

Are you willing to make the digital shift to be a model-driven business?

If you enjoyed this episode, check out show notes, resources, and more at www.superdatascience.com/212

The Super Data Science Podcast: Big Data | Analytics Careers | Mentors | Success

SDS 212: Model Driven Vs Data Driven

When data scientists use a linear regression to look for causal relationships between a treatment and an outcome, what they’re usually finding is the so-called average treatment effect. In other words, on average, here’s what the treatment does in terms of making a certain outcome more or less likely to happen. But there’s more to life than averages: sometimes the relationship works one way in some cases, and another way in other cases, such that the average isn’t giving you the whole story. In that case, you want to start thinking about heterogeneous treatment effects, and this is the podcast episode for you.

Relevant links:
https://eng.uber.com/analyzing-experiment-outcomes/
https://multithreaded.stitchfix.com/blog/2018/11/08/bandits/
https://www.locallyoptimistic.com/post/against-ab-tests/

Heterogeneous Treatment Effects

Why do we pronounce data with a capital D?

Understanding Data
by Cassie Kozyrkov.

Read by the author.

Text version: bit.ly/quaesita_hist
Blog: medium.com/@kozyrkov
Videos: bit.ly/statthinking
Music by my friends Raised By Dinosaurs. Full song: bit.ly/dipod_song

Understanding Data

Software engineers, architects and team leads have found inspiration to drive change and innovation in their team by listening to the weekly InfoQ Podcast. They have received essential information that helped them validate their software development map. We have achieved that by interviewing some of the top CTOs, engineers and technology directors from companies like Uber, Netflix and more. Over 1,200,000 downloads in the last 3 years.

Streaming: Danny Yuan on Real-Time, Time Series Forecasting @Uber

InfoQ

Recent comments

Anshul Gupta 32

Related tracks