Name: Ep 23 - The Three Eras Of Data Engineering by DataAware by Ascend
Duration: 36 min
Description: Sean and Paul talk the three eras of data engineering teams move through as they get more mature with data processing. We unpack the kinds of metadata required at each stage, and how realistic it is to build a system that processes data in incremental packets instead of full reductions.

Sean and Paul talk about the impact that Generative AI will have on data engineering. They explore similarities between LLMs and data pipeline automation models, and review common components of an automation stack.

Ep 22 - Generative AI In Data Engineering

Sean and Paul talk the three eras of data engineering teams move through as they get more mature with data processing. We unpack the kinds of metadata required at each stage, and how realistic it is to build a system that processes data in incremental packets instead of full reductions.

Ep 23 - The Three Eras Of Data Engineering

Join Sean and Paul as they unpack the trends from Ascend’s annual DataAware Pulse Survey. Learn why executives and individual contributors disagree on strategy so often... and why many in the data team want to drive automation but struggle to achieve it. All that and a full recap of the 2023 Big Data London event in this episode!

Ep 24 - 2023 Data Engineering Trends

Join Sean and Paul as they talk about how an intelligent data pipeline controller brings traditional developer techniques to the world of data engineering. Learn about Sean’s framework for deciding how frequently you should refresh your pipelines and see if streaming and batch data processing are converging. Plus, what is the definition of real-time? All this and more in this week’s episode!

EP 25 - The Benefits Of Data Pipeline Automation

Are data pipelines a new security concern for LLM projects? Join Sean and Paul as they unpack the recent AI safety and security guidelines unveiled by the US and UK governments, endorsed by 16 other countries and dozens of top AI companies in Silicon Valley.

Ep 26 - AI Security Guidelines

DataAware by Ascend is a community-focused podcast to discuss all things data engineering.

With a variety of guests from all facets of data engineering and associated teams, each DataAware episode will look in-depth at the role of data engineering and data teams, trends, best (and worst) practices, real world use cases, and more.

Ep 23 - The Three Eras Of Data Engineering

DataAware by Ascend

Related tracks