In today's episode Thomas Betts talks to Lily Mara, Engineering Manager at OneSignal in San Mateo, California. She manages the infrastructure services team, which is responsible for in-house services used by other OneSignal engineering teams. They will discuss how to build a reliable Kafka data processing pipeline.

Read a transcript of this interview: https://bit.ly/3ONTM63

Subscribe to the Software Architects’ Newsletter [monthly]: www.infoq.com/software-architect…mpaign=architectnl

Upcoming Events:

QCon San Francisco
https://qconsf.com/ 
Oct 2-6, 2023

QCon London
https://qconlondon.com/ 
April 8-10, 2024

Follow InfoQ:
- Mastodon: https://techhub.social/@infoq
- Twitter: twitter.com/InfoQ
- LinkedIn: www.linkedin.com/company/infoq
- Facebook: bit.ly/2jmlyG8
- Instagram: @infoqdotcom
- Youtube: www.youtube.com/infoq

Write for InfoQ
- Join a community of experts.
- Increase your visibility.
- Grow your career.
www.infoq.com/write-for-infoq/?u…aign=writeforinfoq

InfoQ Podcast

Transcript

Building a Reliable Kafka Data Processing Pipeline With Lily Mara

Jonas Bonér, CTO of LightBend and creator Akka, discusses using Akka when developing distributed systems. He talks about the Actor Model, and how every Microservice needs to be viewed as a system to be successful. 

Why listen to this podcast:

 - Akka is JVM-based framework design for developing distributed systems leveraging the Actor Model - an approach for writing concurrent systems that treat actors as universal primitives and the most successful model with abstraction has been streaming
 - Circuit breakers in Akka are a backup and retry policy; they protect you by capturing failure data and allow you to roll back 
 - Every Microservice needs to be viewed as a system, it needs to have multiple parts that run on different machines in order to function and be fully resilient - thus is a Microsystem
 - Two different trends have emerged when it comes to hardware and environments: one is the trend toward Multi-core, the is a movement toward virtualized environments and the cloud
 - Saga pattern of managing long running transactions in a distributed system fits very well with messaging style architectures

Notes and links can be found on: http://bit.ly/2kwB2eB

Akka
The Actor Model
When Akka and the Actor Model is the perfect choice
Circuit breakers patterns in distributed systems
Two trends toward Multi-core
Reactive Manifesto
Event Driven vs. Message Driven
Reactive Programming and Streams
Microliths to Microsystems
What do you have to get right before you start trying to deploy a distributed systems?
Working with ML / AI at Lightbend to understand tracing through distributed system
Saga Pattern

More on this:
Quick scan our curated show notes on InfoQ http://bit.ly/2kwB2eB

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Show Notes

Jonas Bonér on the Actor Model, Akka, Reactive Programming, Microservices and Distributed Systems

Software engineers, architects and team leads have found inspiration to drive change and innovation in their team by listening to the weekly InfoQ Podcast. They have received essential information that helped them validate their software development map. We have achieved that by interviewing some of the top CTOs, engineers and technology directors from companies like Uber, Netflix and more. Over 1,200,000 downloads in the last 3 years.

Building a Reliable Kafka Data Processing Pipeline With Lily Mara

InfoQ

Related tracks