infoq

ml.platforms

Get a handle on what it means to make proper use of streaming data, what tools to use and how to use them best by learning from the people that have already done it. Listen to our dedicated podcast series.

Summer series- Streaming

podcast

In this week’s podcast, Thomas Betts talks with Chris Richardson, a developer, architect, Java Champion and author of POJOs in Action. Before his workshop on Microservices w/ Spring Boot and Docker at QCon San Francisco 2016, Richardson took time to discuss his ideas on how to use DDD and CQRS concepts as a guide for implementing a robust microservices architecture.

Why listen to this podcast:

 - "Microservice architecture" is a better term than "microservices".   The latter suggests that a single microservice is somehow interesting
 - The concepts discussed in Domain-Driven Design are an excellent guide for how to implement a microservices architecture
 - Bounded Contexts correspond well to individual microservices
 - Event sourcing and CQRS provide patterns for how to implement loosely coupled services
 - When converting a monolith to microservices, avoid a big bang rewrite, in favor of an iterative approach

Notes and links can be found on: http://bit.ly/2hZ8TM1

11m:51s - Microservices must be loosely coupled, usually creating a model with one database per service.
12m:45s - There is a business requirement to maintain data consistency across services, and using an event driven architecture is a good way to achieve that.
13m:38s - Event sourcing is specific technique for persisting domain objects as a series of events.
14m:11s - Just as transactions don’t like to be split across microservices, queries cannot simply join across multiple data sources. CQRS provides a solution that accommodates querying via microservices and materialized views.

More on this:
Quick scan our curated show notes on InfoQ. http://bit.ly/2hZ8TM1

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

InfoQ Podcast

Show Notes

Chris Richardson on Domain-Driven Microservices Design

In this week’s podcast Wes Reisz talks to Neha Batra, a software engineer at Pivotal Labs. Neha spoke about pair programming in her recent QCon San Francisco 2016 presentation, and has taken time to discuss techniques to get started with the practice as well as tips for implementing it on your team. Neha also touches on vulnerability based trust and how it can help effectively build a trusting team environment.
Why listen to this podcast:

 - If you successfully start with pair programming, other tenants of XP are pulled along with you
 - Ways to get creative with remote pairing to make it work
 - The daily retro
 - Overcoming hesitance with managers when trying to implement pair programming full time
 - Vulnerability based trust building

Notes and links can be found on: http://bit.ly/2i2a0sJ

How has Pair Programming Evolved Over the Years? 

6m:17s - A lot of the fundamentals are the same, but with XP we take it to the extreme to be able to do it  eight hours a day.
6m:24s - To pair for eight hours a day we adapt to a lot of other process to create a simpler way of working, giving us an easier level to default to.
6m:44s - We use phrases in the team to make sure we agree on a test, that there are no false positives, when to refactor etc. This helps us avoid accruing code debt since we don’t do code reviews.

More on this:
Quick scan our curated show notes on InfoQ. http://bit.ly/2i2a0sJ

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Neha Batra - Pivotal Labs Pair Programming

Wesley Reisz talks to Slava Oks, who has worked at Microsoft for over 20 years on flagship products, including SQL Server. He also led the kernel team who worked on the Midori operating system. More recently, he has worked on bringing SQL Server to Linux.

Why listen to this podcast:

 - Microsoft SQL Server runs on Linux through a containerised approach called Drawbridge
 - Drawbridge implements a Linux loader and a minimal set of ABI calls to allow an in-process NT user mode kernel to run
 - SQL Server runs on top of a SQL platform layer (called SQL OS) that could be ported to run on Drawbridge
 - SQL Server had supportability commands added to allow the state of the system to be measured with SQL calls
 - A number of efficiency gains were applied to both the Drawbridge components and the SQL Server code to bring performance to within 20% of the equivalent process running on Windows

Notes and links can be found on: http://bit.ly/2kVfatX

Drawbridge
ABIs
Security
LibOS
Linux
SQL OS
Supportability
Some assembly required
Addressing
Performance

More on this:
Quick scan our curated show notes on InfoQ http://bit.ly/2kVfatX

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Architecting SQL Server on Linux: Slava Oks on Drawbridge, LibOS, & Addressing Between Windows/Linux

Wesley Reisz talks to Daniel Bryant on moving from monoliths to micro-services, covering bounded contexts, when to break up micro-services, event storming, practices like observability and tracing, and more.

Why listen to this podcast:

- Migrating a monolith to micro-services is best done by breaking off a valuable but not critical part first.
 - Designing a greenfield application as micro-services requires a strong understanding of the domain.
 - When a request enters the system, it needs to be tagged with a correlation id that flows down to all fan-out service requests.
 - Observability and metrics are essential parts to include when moving micro-services to production.
 - A service mesh allows you to scale services and permit binary transports without losing observability.

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2pFYBiT

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2pFYBiT

Daniel Bryant on Microservices and Domain Driven Design

On the InfoQ Podcast this week, Wes Reisz talks with the Queen of Patterns, Linda Rising. Linda discusses her thoughts on the importance of patterns, she answers questions about what really is a pattern, and how she became involved in working with them. Throughout the podcast she discusses a variety of organizational and personal patterns and finally wraps with patterns to apply when driving change and innovation.

Why listen to this podcast:

 - You have to realise that there’s nothing you can do about other people. The only person you can affect is yourself.
 - A pattern is not a band-aid that you use once. You use it in a context where you use it in conjunctions with other patterns.
 - Take baby steps when driving change in an organisation, and seek out a pocket of receptive people to drive it.
 - Slack is an important part to have in life, so that if something comes along you can absorb it without having to stop doing something else.
 - Listen, Listen, Listen.

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2vLIsMC

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2vLIsMC

Linda Rising on the Importance of Patterns, Her Journey, & Patterns for Driving Change/Innovation

In this podcast, we are talking to Tyler Akidau, a senior engineer at Google, who leads the technical infrastructure and data processing teams in Seattle, and a founding member of the Apache Beam PMC and a passionate voice in the streaming space. This podcast will cover data streaming and the 2015 DataFlow Model streaming paper [http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf] and much of the concepts covered, such as why dealing with out-of-order data is important, event time versus processing time, windowing approaches, and finally preview the track he is hosting at QConf SF next week.

Why listen to this podcast:

 - Batch processing and streaming aren’t two incompatible things; they are a function of different windowing options.
 - Event time and processing time are two different concepts, and may be out of step with each other.
 - Completeness is knowing that you have processed all the events for a particular window.
 - Windowing choice can be answered from the what, when, where, how questions.
 - Unbounded versus bounded data is a better dimension than stream or batch processing.

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2AyBTAb

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2AyBTAb

Apache Beam Founder Tyler Akidau Discusses Streaming System and Their Complexities

In this podcast, Deam Wampler discusses fast data, streaming, microservices, and the paradox of choice when it comes to the options available today building data pipelines.

Why listen to this podcast:

* Apache Beam is fast becoming the de-facto standard API for stream processing
* Spark is great for batch processing, but Flink is tackling the low-latency streaming processing market
* Avoid running blocking REST calls from within a stream processing system - have them asynchronously launched and communicate over Kafka queues
* Visibility into telemetry of streaming processing systems is still a new field and under active development
* Running the fast data platform is easily launched on an existing or new Mesosphere DC/OS runtime

More on this: Quick scan our curated show notes on InfoQ http://bit.ly/2BYTMbI

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Want to see extented shownotes? Check the landing page on InfoQ: http://bit.ly/2BYTMbI

Fast Data with Dean Wampler

On this week’s podcast, Danny Yuan, Uber’s Real-time Streaming/Forecasting Lead, lays out a thorough recipe book for building a real-time streaming platform with a major focus on forecasting. In this podcast, Danny discusses everything from the scale Uber operates at to what the major steps for training/deploy models in an iterative (almost Darwinistic) fashion and wraps with his advice for software engineers who want to begin applying machine learning into their day-to-day job.  

Why listen to this podcast:

* Uber processes 850,000 - 1.3 million messages per second in their streaming platform with about 12 TB of growth per day. The system’s queries scan 100 million to 4 billion documents per second.
* Uber’s frontend is mobile. The frontend talks to an API layer. All services generate events that are shuffled into Kafka. The real-time forecasting pipeline taps into Kafka to processes events and stores the data into Elasticsearch. * There is a federated query layer in front of Elasticsearch to provide OLAP query capabilities.
* Apache Flink’s advanced windowing features, programming model, and checkpointing convinced Uber to move away from the simplicity of Apache Samza.
* The forecasting system allows Uber to remove the notion of delay by using recent signals plus historical data to project what is happening now and what will happen into the future.
* Uber’s pipeline for deploying ML models: HDFS, feature engineering, organizing into data structures (similar to data frames), deploy mostly offline training models, train models, & store into a container-based model manager. 
* A model serving layer is used to pick which model to use, forecasting results are stored in an OLAP data store, a validation layer compares real results against forecast results to verify the model is working as desired, and a rollback feature enables poor performing models to be automatically replaced by previous one.
* “Without output, you don’t have input.” If you want to start leveraging machine learning, developers just need to start doing. Start with intuition and practice. Over time ask questions and learn what you need, then apply a laser focus to gain that knowledge.

You can also subscribe to the InfoQ newsletter to receive weekly updates on the hottest topics from professional software development. bit.ly/24x3IVq

Subscribe: www.youtube.com/infoq
Like InfoQ on Facebook: bit.ly/2jmlyG8
Follow on Twitter: twitter.com/InfoQ
Follow on LinkedIn: www.linkedin.com/company/infoq
Check the landing page on InfoQ: https://bit.ly/2GJQbUo

Streaming: Danny Yuan on Real-Time, Time Series Forecasting @Uber

In this episode of the Data Show, I spoke with Ashok Srivastava, senior vice president and chief data officer at Intuit. He has a strong science and engineering background, combined with years of applying machine learning and data science in industry. Prior to joining Intuit, he led the teams responsible for data and artificial intelligence products at Verizon. I wanted his perspective on a range of issues, including the role of the chief data officer, ethics in machine learning, and the emergence of AI technologies for enterprise products and applications.

The Real Value of Data Requires a Holistic View of the End- to-End Data Pipeline

In this episode of the Data Show, I spoke with Harish Doddi, co-founder and CEO of Datatron, a startup focused on helping companies deploy and manage machine learning models. As companies move from machine learning prototypes to products and services, tools and best practices for productionizing and managing models are just starting to emerge. Today’s data science and data engineering teams work with a variety of machine learning libraries, data ingestion, and data storage technologies. Risk and compliance considerations mean that the ability to reproduce machine learning workflows is essential to meet audits in certain application domains. And as data science and data engineering teams continue to expand, tools need to enable and facilitate collaboration.

As someone who specializes in helping teams turn machine learning prototypes into production-ready services, I wanted to hear what Doddi has learned while working with organizations that aspire to “become machine learning companies.”

Simplifying Machine Learning Lifecycle Management

In this episode of the Data Show, I spoke with Eric Jonas, a postdoc in the new Berkeley Center for Computational Imaging. Jonas is also affiliated with UC Berkeley’s RISE Lab. It was at a RISE Lab event that he first announced Pywren, a framework that lets data enthusiasts proficient with Python run existing code at massive scale on Amazon Web Services. Jonas and his collaborators are working on a related project, NumPyWren, a system for linear algebra built on a serverless architecture. Their hope is that by lowering the barrier to large-scale (scientific) computation, we will see many more experiments and research projects from communities that have been unable to easily marshal massive compute resources. We talked about Bayesian machine learning, scientific computation, reinforcement learning, and his stint as an entrepreneur in the enterprise software space.

Building Accessible Tools for Large-Scale Computation and Machine Learning

In this episode of the Data Show, I spoke with Vitaly Gordon, VP of data science and engineering at Salesforce. As the use of machine learning becomes more widespread, we need tools that will allow data scientists to scale so they can tackle many more problems and help many more people. We need automation tools for the many stages involved in data science, including data preparation, feature engineering, model selection and hyperparameter tuning, as well as monitoring.

I wanted the perspective of someone who is already faced with having to support many models in production. The proliferation of models is still a theoretical consideration for many data science teams, but Gordon and his colleagues at Salesforce already support hundreds of thousands of customers who need custom models built on custom data. They recently took their learnings public and open sourced TransmogrifAI, a library for automated machine learning for structured data, which sits on top of Apache Spark.

Building Tools for Enterprise Data Science

Software engineers, architects and team leads have found inspiration to drive change and innovation in their team by listening to the weekly InfoQ Podcast. They have received essential information that helped them validate their software development map. We have achieved that by interviewing some of the top CTOs, engineers and technology directors from companies like Uber, Netflix and more. Over 1,200,000 downloads in the last 3 years.