DataPodcast

Jonas Bonér, CTO of LightBend and creator Akka, discusses using Akka when developing distributed systems. He talks about the Actor Model, and how every Microservice needs to be viewed as a system to be successful. 

Why listen to this podcast:

 - Akka is JVM-based framework design for developing distributed systems leveraging the Actor Model - an approach for writing concurrent systems that treat actors as universal primitives and the most successful model with abstraction has been streaming
 - Circuit breakers in Akka are a backup and retry policy; they protect you by capturing failure data and allow you to roll back 
 - Every Microservice needs to be viewed as a system, it needs to have multiple parts that run on different machines in order to function and be fully resilient - thus is a Microsystem
 - Two different trends have emerged when it comes to hardware and environments: one is the trend toward Multi-core, the is a movement toward virtualized environments and the cloud
 - Saga pattern of managing long running transactions in a distributed system fits very well with messaging style architectures

Notes and links can be found on: http://bit.ly/2kwB2eB

Akka
The Actor Model
When Akka and the Actor Model is the perfect choice
Circuit breakers patterns in distributed systems
Two trends toward Multi-core
Reactive Manifesto
Event Driven vs. Message Driven
Reactive Programming and Streams
Microliths to Microsystems
What do you have to get right before you start trying to deploy a distributed systems?
Working with ML / AI at Lightbend to understand tracing through distributed system
Saga Pattern

More on this:
Quick scan our curated show notes on InfoQ http://bit.ly/2kwB2eB

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

InfoQ Podcast

Show Notes

Jonas Bonér on the Actor Model, Akka, Reactive Programming, Microservices and Distributed Systems

Vaughn Vernon is thought-leader in the space of reactive software and Domain Driven Design (DDD). Vaughn has recently released a new open source project called vlingo. The platform is designed to support DDD at the framework and toolkit level. On today’s podcast, Vaughn discusses what the framework is all about, why he felt it was needed, and some of the design decisions made in developing the platform, including things like the architecture, actor model decisions, clustering algorithm, and how DDD is realized with the framework.

Why listen to this podcast:

 - Vlingo is an open source system for building distributed, concurrent, event-driven, reactive microservices that supports (at the framework level) Domain Driven Design.
 - The platform is in the early stages. It runs on the JVM. There is a port to C#. All code is pushed up stream.
 - The platform uses the actor model and all messages are sent in a type-safe way.
 - Vlingo supports clustering and uses a bully algorithm to achieve consensus.

More on this: Quick scan our curated show notes on InfoQ https://bit.ly/2MwEWxb

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Check the landing page on InfoQ: https://bit.ly/2MwEWxb

Vaughn Vernon on Developing a Domain Driven Design first Actor-Based Microservices Framework

In June of this year, Consul 1.2 was released. The release expanded Consul’s capability around service segmentation (controlling who and how services connect East and West). On this week’s podcast, Wes and Mitchell discuss Consul in detail. The two discuss Consul’s design decisions around focusing on user space networking, layer 4 routing, Go, Windows’ performance characteristics, the roadmap for eBPF on Linux, and an interesting feature that Consul implements called Network Tomography. The show wraps with Mitchell’s discussion on some of the research that Hashicorp is doing around machine learning and security with Consul.

Why listen to this podcast:

- Consul is first and foremost a centralized service registry that provides discovery. While it has a key-value store, it is Consul’s least important feature.
With the June release (1.2), Consul entered more into the space of a service mesh with the focus on service segmentation (controlling how you connect and who can connect).
- Hashicorp attempts to limit the language fragmentation in the Company and has seen a lot of success leveraging Go across their platforms. Therefore, Consul is written in Go.
- Because Consul focused on layer 4 first, it is recommended to leverage the recent integration with Envoy for achieving high degrees of observability.
- All of the network routing with Consul happens in user space at this point; however, kernel space routing with eBPF is planned for the near term. The focus, at this point, is safely cross-compiling to every platform and addressing the most possible use cases. The focuses isn’t on the high performance use cases (yet).
- For any two servers across the globe in different data centers, instantly Consul can give you 99th percentile round-trip time between with uses a feature called Network Tomography.

More on this: Quick scan our curated show notes on InfoQ https://bit.ly/2S3ZiSx

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Check the landing page on InfoQ: https://bit.ly/2S3ZiSx

Mitchell Hashimoto on Consul since 1.2 and its Role as a Modern Service Mesh

Josh Wills, a software engineer working on data engineering problems at Slack, discusses the Slack data architecture and how they build and observe their pipelines. Josh, along with color commentary such as the move from IC to manager (and back), discusses recommendations, tips, tools, and lessons Slack engineering teams discovered while building products like Slack Search. The podcast covers machine learning, observability, data engineering, and general practices for building highly resilient software.

Why listen to this podcast:

 - Slack has a philosophy of building only what they need. They have a don’t reinvent the wheel mindset. 
 - Slack was originally a PHP monolith. Today, it is largely Hack-lang, HHVM, and several Java and Go binarys. On the data side, application logs are in Thrift (there is a plan to migrate to protobuf). Events are processed through a Kafka cluster that handles 100,000s of events per second. Everything is kept in S3 with a large Hive metastore. EMR is spun up on demand. Presto, Airflow, Slack, Snowflake (business analytics), Quiver (key value store) are all used. 
 - ML worked best for Slack when it was used to help people answer questions. Things like Learn to Rank (LTR) become the most effective use of ML for Slack.
 - You can get pretty far with rules. Use machine learning when that’s all that’s left. 
 - When you start applying observability to your data pipeline, a key lesson for Slack was to really focus on structured data, tracing, high cardinality events. This let them really use the tools they were already familiar with (ELK, Prometheus, Grafana) and go deep into understanding what’s happening in the systems.

More on this: Quick scan our curated show notes on InfoQ https://bit.ly/2PsVA4q
You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Check the landing page on InfoQ: https://bit.ly/2PsVA4q

Josh Wills on Building Resilient Data Engineering and Machine Learning Products at Slack

In this podcast, Daniel Bryant sat down with Zhamak Dehghani, principal consultant, member of technical advisory board, and portfolio director at ThoughtWorks. Topics discussed included: the motivations for becoming a data-driven organization; the challenges of adapting legacy data platforms and ETL jobs; and how to design and build the next generation of data platforms using ideas from domain-driven design and product thinking, and modern platform principles such as self-service workflows.

Why listen to this podcast:

 - Becoming a data-driven organization remains one of the top strategic goals of many organizations. Being able to rapidly run experiments and efficiently analyse the resulting data can provide a competitive advantage.
 - There are several “architecture failure modes” within existing enterprise data platforms. They are centralized and monolithic. The composition of data pipelines are often highly-coupled, meaning that a change to the data format will require a cascade of changes throughout the pipeline. And finally, the ownership of data platforms is often siloed and hyper-specialized.
 - The next generation of enterprise data platform architecture requires a paradigm shift towards ubiquitous data with a distributed data mesh.
Instead of flowing the data from domains into a centrally owned data lake or platform, domains need to host and serve their domain datasets in an easily consumable way.
 - Domain data teams must apply product thinking to the datasets that they provide; considering their data assets as their products, and the rest of the organization's data scientists, ML and data engineers as their customers.
The key to building the data infrastructure as a platform is (a) to not include any domain specific concepts or business logic, keeping it domain agnostic, and (b) make sure the platform hides all the underlying complexity and provides the data infrastructure components in a self-service manner.


More on this: Quick scan our curated show notes on InfoQ https://bit.ly/39exTWl
You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Check the landing page on InfoQ: https://bit.ly/39exTWl

Zhamak Dehghani on Data Mesh, Domain-Oriented Data, and Building Data Platforms

Charles Humble talks to Akhilesh Gupta, the technical lead for LinkedIn's real-time delivery infrastructure, and also LinkedIn messaging. They discuss the architecture behind LinkedIn’s real-time platform, its building blocks, the frameworks used and other technical details.

Listen to the podcast for more.

Curated transcript and more information on the https://bit.ly/2DujU3U
Follow us on Facebook, Twitter, LinkedIn, Youtube: @InfoQ 
Follow us on Instagram: @infoqdotcom

Stay informed on emerging trends, peer-validated early adoption of technologies, and architectural best practices. Subscribe to The Software Architects’ Newsletter: www.infoq.com/software-architects-newsletter/

Transcript

Akhilesh Gupta on the Architecture of LinkedIn’s Real-time Messaging Platform

In this podcast, John DesJardins, field CTO and VP solution architecture at Hazelcast, sat down with InfoQ podcast co-host Daniel Bryant. Topics discussed included: how in-memory data grids have evolved, use cases at the edge (IoT, ML inference), integration of stream processing APIs and techniques, and how data grids can be used within application modernization.

Listen to the podcast for more.

Curated transcript and more information on the https://bit.ly/2FtBIgp
Follow us on Facebook, Twitter, LinkedIn, Youtube: @InfoQ 
Follow us on Instagram: @infoqdotcom

Stay informed on emerging trends, peer-validated early adoption of technologies, and architectural best practices. Subscribe to The Software Architects’ Newsletter: www.infoq.com/software-architects-newsletter/

John DesJardins on In-Memory Data Grids, Stream Processing, and App Modernization

Software engineers, architects and team leads have found inspiration to drive change and innovation in their team by listening to the weekly InfoQ Podcast. They have received essential information that helped them validate their software development map. We have achieved that by interviewing some of the top CTOs, engineers and technology directors from companies like Uber, Netflix and more. Over 1,200,000 downloads in the last 3 years.

Josh Wills on Building Resilient Data Engineering and Machine Learning Products at Slack

InfoQ

Related tracks