Vivek Yadav, an engineering manager from Stripe, shares his experience in building a testing system based on multi-year worth of data. He shares insights into why Apache Spark was the choice for creating such a system and how it fits in the "traditional" engineering practices.

Read a transcript of this interview: https://bit.ly/4o08NjD

Subscribe to the Software Architects’ Newsletter for your monthly guide to the essential news and experience from industry peers on emerging patterns and technologies:
https://www.infoq.com/software-architects-newsletter 

Upcoming Events:

QCon AI New York 2025 (December 16-17, 2025)
https://ai.qconferences.com/

QCon London 2026 (March 16-19, 2026)
QCon London equips senior engineers, architects, and technical leaders with trusted, practical insights to lead the change in software development. Get real-world solutions and leadership strategies from senior software practitioners defining current trends and solving today's toughest software challenges.
https://qconlondon.com/

QCon AI Boston 2026 (June 1-2, 2026)
Learn how real teams are accelerating the entire software lifecycle with AI.
https://boston.qcon.ai


The InfoQ Podcasts: 
Weekly inspiration to drive innovation and build great teams from senior software leaders. Listen to all our podcasts and read interview transcripts:

- The InfoQ Podcast https://www.infoq.com/podcasts/ 
- Engineering Culture Podcast by InfoQ https://www.infoq.com/podcasts/#engineering_culture 
- Generally AI: https://www.infoq.com/generally-ai-podcast/ 

Follow InfoQ:
- Mastodon: https://techhub.social/@infoq 
- X: https://x.com/InfoQ?from=@ 
- LinkedIn: https://www.linkedin.com/company/infoq/
- Facebook: https://www.facebook.com/InfoQdotcom#
- Instagram: https://www.instagram.com/infoqdotcom/?hl=en
- Youtube: https://www.youtube.com/infoq
- Bluesky: https://bsky.app/profile/infoq.com


Write for InfoQ: Learn and share the changes and innovations in professional software development.
- Join a community of experts.
- Increase your visibility.
- Grow your career.
https://www.infoq.com/write-for-infoq

Transcript

How to Use Apache Spark to Craft a Multi-Year Data Regression Testing and Simulations Framework

Software engineers, architects and team leads have found inspiration to drive change and innovation in their team by listening to the weekly InfoQ Podcast. They have received essential information that helped them validate their software development map. We have achieved that by interviewing some of the top CTOs, engineers and technology directors from companies like Uber, Netflix and more. Over 1,200,000 downloads in the last 3 years.

New tracks tagged #Big-Data Systems