pipelines

In this episode, we dove into the relationship between Airflow and Kuberenetes and interviewed Daniel Imberman, Senior Software Engineer at Bloomberg (1:30), and Greg Neiheisel, CTO here at Astronomer (37:31). Daniel has done most of the work on the Kubernetes executor for Airflow and Greg plans to take on a chunk of the development going forward, so it was really interesting to hear both of their perspectives on the project. Enjoy!

Airflow on Kubernetes

Data pipelines become chaotic with pressures of agile, democratization, self-service, and organizational “pockets” of analytics. From enterprise BI to self-service analysis, data pipeline management should ensure analysis results are traceable, reproducible, and of production strength. Robust data pipelines rely on eight critical components.

Originally published at https://www.eckerson.com/articles/the-complexities-of-modern-data-pipelines

pipelines

Felipe Quirce