In this episode, we speak with Dr. Jon Krohn about the life cycle of open-source LLMs. Jon is a co-founder and chief data scientist at the machine learning company Nebula. He is the author of the book Deep Learning Illustrated, which was an instant #1 bestseller and was translated into seven languages. He is also the host of the fabulous SuperDataScience podcast, the data science industry’s most listened-to podcast. An incredible instructor and speaker, Jon’s workshops at ODSC conferences and other events are always one of our most popular.

Topics:
1. Guest Introduction
2. Definition of an open source LLMs and what it means to be truly open source
3. The importance of LLM weights and neural networks architecture for training
4. Transformer architecture
5. Apple expanding their AI team
6. What do I need to train or fine-tune an LLM
7. Key libraries for fine-tunning an LLM
8. The LoRA (Low-Rank Adaptation) technique for efficiently fine-tuning large language models
9. Testing and evaluating LLMs prior to deploying in production
10. Retrieval Augmented Generation (RAG)
11. Deploying LLM to production
12. How to keep inference costs down
13. How can people follow Jon’s content (see show notes also)
Show Notes:
More about Jon:
LinkedIn - https://www.linkedin.com/in/jonkrohn/
Jon’s YouTube Channel - https://www.youtube.com/c/jonkrohnlearns
Jon’s Monthly Newsletter - https://www.jonkrohn.com/

Tools and Resources:
Michael Nielsen's eBook on Neural Networks and Deep Learning - http://neuralnetworksanddeeplearning.com/
PyTorch Lightning is the deep learning framework - https://lightning.ai/docs/pytorch/stable/

Hugging Face Transformers Library - https://huggingface.co/docs/transformers/v4.17.0/en/index

Vicuna: An Open-Source Chatbot - https://lmsys.org/blog/2023-03-30-vicuna/

LoRA: Low-Rank Adaptation of Large Language Models - https://arxiv.org/abs/2106.09685

SDS 674: Parameter-Efficient Fine-Tuning of LLMs using LoRA (Low-Rank Adaptation) - https://www.superdatascience.com/podcast/parameter-efficient-fine-tuning-of-llms-using-lora-low-rank-adaptation

Unsloth for finetuning Llama 3, Mistral & Gemma - https://github.com/unslothai/unsloth

Phoenix: an open-source AI Observability & Evaluation tool - https://github.com/Arize-ai/phoenix

ODSC Podcast with Amber Roberts on Phoenix and troubleshooting LLMs - https://www.odsc.com/podcast/#e33

Weights & Biases - https://wandb.ai/site
This episode was sponsored by:  
Ai+ Training https://aiplus.training/ 
Home to hundreds of hours of on-demand, self-paced AI training, ODSC interviews, free webinars, and certifications in in-demand skills like LLMs and Prompt Engineering

And created in partnership with ODSC https://odsc.com/ 
The Leading AI Training Conference, featuring expert-led, hands-on workshops, training sessions, and talks on cutting-edge AI topics and

Never miss an episode, subscribe now!

Training and Deploying Open-Source LLMs with Dr. Jon Krohn

In this episode of the ODSC AIX Podcast, host Sheamus McGovern speaks with Arvind Narayanan, professor of computer science at Princeton University and co-author of the widely acclaimed book AI Snake Oil. Arvind is also the director of the Center for Information Technology Policy and one of TIME's 100 most influential people in AI.
Together, they unpack the growing disconnect between AI hype and practical deployment, focusing particularly on agentic AI systems and why current evaluation methods are falling short. Arvind discusses the moral limits of predictive AI, the myth of superintelligence, and how practitioners can ground their optimism in accountability, transparency, and more realistic expectations.

Key Topics Covered
- The origins and impact of AI Snake Oil
- Why many AI systems are overhyped or misapplied
- Predictive vs. generative AI: what's really at stake
- Why superintelligence is a myth—and a distraction
- The flawed assumption behind “upstream” evaluations
- Engineering and ethical limitations of AI agents in real-world deployment
- Misinformation, misuse, and human responsibility in the AI age
- The importance of resilience over restriction in AI governance

Memorable Outtakes
💬 “We came to the conclusion that superintelligence is a bit of a myth. AI will continue to improve, but …”
💬 “Broken AI is often appealing to broken institutions. You can’t fix a flawed hiring process with more automation—you need more humanity.”
💬 "Upstream evaluation is mostly dead."

References & Resources
- Arvind Narayanan Academic Profile: https://www.cs.princeton.edu/~arvindn/
- AI Snake Oil (Book & Newsletter)  https://www.aisnakeoil.com
- “How to Recognize AI Snake Oil” (Slides)
 https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
- AI Agents That Matter (Research Paper) https://arxiv.org/abs/2407.01502
- Holistic Agent Leaderboard (HAL)   https://hal.institutions.ai
- Essay: “AGI Is Not a Milestone”
 https://aisnakeoil.substack.com/p/agi-is-not-a-milestone
- Essay: “AI as Normal Technology”
 https://aisnakeoil.substack.com/p/ai-as-normal-technology
- HAL: Holistic Agent Leaderboard  https://hal.cs.princeton.edu/
- Agentic AI Summit
 https://www.summit.ai/
- ODSC West –  https://odsc.com/california
- Arvind’s ODSC East Keynote Session, AI Agents: Taking Stock and Looking Ahead: https://odsc.com/speakers/ai-agents-taking-stock-and-looking-ahead/

Sponsored by:
Agentic AI Summit 2025
 Join the premier virtual event for AI builders from July 15–30. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
 🔥 Use code podcast for 10% off any ticket.
 Register now: https://www.summit.ai/
ODSC West 2025 – The Leading AI Training Conference
 Attend in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
 🔥 Use code podcast for 10% off any ticket.
 Learn more: https://odsc.com/california

The AI Superintelligence Myth with Arvind Narayanan

In this episode of the ODSC Aix Podcast, host Sheamus McGovern speaks with Dr. Regina Barzilay, a Distinguished Professor of AI and Health at MIT, and one of the leading voices in applying artificial intelligence to real-world medical challenges. Dr. Barzilay is also the AI faculty lead at MIT’s Jameel Clinic, where her work spans early disease detection, personalized treatment, and AI-powered drug discovery.
A recipient of the MacArthur “Genius” Fellowship, the AAAI Squirrel AI Award, and a member of both the National Academy of Engineering and the National Academy of Medicine, Dr. Barzilay has built a career bridging state-of-the-art AI with the pressing needs of patients and clinicians.
Together, they explore why AI tools that can accurately predict cancer and other diseases are still rarely used in clinical practice—and what it will take to bridge the gap between research and real-world care.
Key Topics Covered
Why most diseases are diagnosed too late—and how AI can detect them before symptoms appear
The story behind MIRAI (for breast cancer) and SYBIL (for lung cancer) predictive models
Why powerful clinical AI tools aren’t widely adopted in hospitals and how the system can change
What it takes to get machine learning models into real-world care across countries and hospitals
The promise of generative AI in drug discovery and the discovery of new antibiotics
Challenges in detecting and treating neurodegenerative diseases like ALS
How AI could enable truly personalized medicine, predicting treatment side effects and responses
The role of AI copilots in clinical workflows and the future of clinician-AI collaboration
Why AI in healthcare must go beyond hype to deliver real, measurable value for patients



Memorable Outtakes
“The technology to save lives is here. We’re just not using it.” — Dr. Regina Barzilay on the gap between research and clinical adoption
“You can’t be an oncologist without using MRI or blood tests—but you can still practice medicine today without AI.” — Dr. Barzilay on structural resistance in healthcare systems
“Imagine giving some of your health data and getting back a real answer: what will happen to you. Not the population. You.” — Dr. Barzilay on personalized risk prediction and treatment modeling
References & Resources
Dr. Regina Barzilay  Academic Profile:https://www.csail.mit.edu/person/regina-barzilay LinkedIn:⁠ https://www.linkedin.com/in/reginabarzilay/⁠
Resources Mentioned in the Episode
MIRAI (AI model for breast cancer risk prediction): ⁠https://jclinic.mit.edu/mirai⁠
SYBIL (AI model for lung cancer risk prediction):  ⁠https://news.mit.edu/2023/ai-model-can-detect-future-lung-cancer-0120⁠
Halicin Antibiotic Discovery: 
MIT Jameel Clinic for Machine Learning in Health:⁠ https://jclinic.mit.edu/⁠
US Preventive Services Task Force (USPSTF):⁠ https://www.uspreventiveservicestaskforce.org/⁠
Professor Dina Katabi’s AI monitoring research (referenced): ⁠https://www.csail.mit.edu/person/dina-katabi
⁠

Sponsored by
Agentic AI Summit 2025 Join the premier virtual event for AI builders from July 15–30. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents. 🔥 Use code podcast for 10% off any ticket. Register now:⁠ https://www.summit.ai/⁠
ODSC West 2025 – The Leading AI Training Conference Attend in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation. 🔥 Use code podcast for 10% off any ticket. Learn more:⁠ https://odsc.com/california⁠

"AI Can Predict Disease—So Why Aren’t Doctors Using It?" with Regina Barzilay

This special episode captures compelling highlights from ODSC East 2025, held May 13–15. Join host Sheamus McGovern as he discusses transformative AI developments with leading experts, researchers, and industry practitioners. From the practical deployment of AI agents and the rising importance of synthetic data to ethical AI strategies and robust risk management practices, this episode offers diverse perspectives shaping the future of AI.

Guests and Key Topics Covered:
- Rob Bailey (CrewAI): Accelerating AI agent adoption in enterprises.
- Ivan Lee (Datasaur): The rise and effectiveness of specialized small language models.
- Tony Kipkemboi (CrewAI): Getting started with AI agents through practical, foundational skills.
- Samuel Colvin (Pydantic): Building robust, type-safe, and observable AI applications.
- Rajiv Shah (Contextual AI): Advanced evaluation methods for generative AI applications.
- Sinan Ozdemir (Author and entrepreneur) Sinan breaks down a comprehensive evaluation of AI agents and multi-agent systems, beyond benchmarks.
- Alexandra Ebert (MOSTLY AI): Ethical AI and synthetic data democratization.
- Kanchana Patlolla (Google Cloud): Overcoming common intelligent agent development and deployment challenges.
- Cal Al-Dhubaib & Lauren Burke-McCarthy (Further): AI risk management, red teaming, and sustainable data science adoption strategies.
- Dr. Andre Franca (Ergodic): Applying causal AI to improve decision-making processes.
- Noah Giansiracusa (Bentley University): Insights into social media algorithms and enhancing user autonomy.
- Allen Downey (PyMC Labs): Practical insights into traditional time series analysis and Bayesian modeling.

Memorable Outtakes:
- "Automation is no longer about wiring APIs together; it’s about launching self-directed agents that negotiate, learn, and act on our behalf." – Rob Bailey, CrewAI
- "Generative AI has turned traditional data science upside down. People forget that you still need rigorous evaluation to align models with the intended problem." – Rajiv Shah, Contextual AI
- "When introducing low-code and no-code platforms, you open the door to build quickly on shaky foundations." – Lauren Burke-McCarthy, Further

References & Resources:
- Rob Bailey (CrewAI): https://www.linkedin.com/in/robmbailey/
- Ivan Lee (Datasaur): https://www.linkedin.com/in/iylee/
- Tony Kipkemboi (CrewAI): https://www.linkedin.com/in/tonykipkemboi/
- Samuel Colvin (Pydantic): https://www.linkedin.com/in/samuel-colvin/
- Rajiv Shah (Contextual AI): https://www.linkedin.com/in/rajistics/
- Sinan Ozdemir (LoopGenius): https://www.linkedin.com/in/sinan-ozdemir/
- Alexandra Ebert (MOSTLY AI): https://www.linkedin.com/in/alexandraebert/
- Kanchana Patlolla (Google Cloud): https://www.linkedin.com/in/kanchanapatlolla/
- Cal Al-Dhubaib (Further): https://www.linkedin.com/in/dhubaib/
- Lauren Burke-McCarthy (Further): https://www.linkedin.com/in/lauren-e-burke/
- Dr. Andre Franca (Ergodic): https://www.linkedin.com/in/francaandre/
- Noah Giansiracusa, PhD (Bentley University): https://www.noahgian.com
- Allen Downey, PhD (PyMC Labs): https://www.linkedin.com/in/allendowney
- Humanity's Last Exam Benchmark:https://agi.safe.ai
- Pydantic: https://docs.pydantic.dev/
- MOSTLY AI Synthetic Data Competition: https://www.mostlyaiprize.com/
- MCP (Model Component Profiling):https://www.anthropic.com/news/model-context-protocol
- Datasaur: https://datasaur.ai/
- CrewAI: https://www.crewai.com/

Sponsored by:
- Agentic AI Summit 2025: Join the premier virtual event for AI builders from July 16–31. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
🔥 Use code podcast for 10% off any ticket.
Register now: https://www.summit.ai/

- ODSC West 2025 – The Leading AI Training Conference
Attend in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
🔥 Use code podcast for 10% off any ticket.
Learn more: https://odsc.com/california

ODSC East 2025 Minisodes

In this episode of the ODSC AIx Podcast, host Sheamus McGovern speaks with Alexandra Ebert, Chief AI and Data Democratization Officer at MOSTLY AI, and one of the foremost voices on privacy-preserving synthetic data and responsible AI. With a diverse background spanning AI ethics, data access policy, and generative AI regulation, Alexandra brings clarity to how synthetic data is not just a privacy tool but a lever for innovation, fairness, and scalable AI adoption.
The discussion explores what synthetic data is (and isn’t), its core advantages and limitations, its role in addressing fairness and access challenges in data-driven organizations, and how practitioners can actively shape better downstream model performance. The episode also dives into the MOSTLY AI Prize—a $100,000 global competition to advance privacy-safe, high-utility synthetic data generation.

Key Topics Covered:
- The different types and use cases of synthetic data (privacy-preserving, simulation-based, creative)
- How synthetic data helps solve the “data access paradox” in regulated industries
- Key advantages and limitations of synthetic data vs. real-world and legacy anonymized data
- Privacy mechanisms: Outlier suppression, statistical mimicry, empirical differential privacy
Real-world use cases in healthcare, finance, telco, and simulation environments
- Fairness-aware synthetic data generation using statistical parity constraints
- Imputing missing data with synthetic distributions
- Agentic AI and the role of synthetic data in enabling secure access layers for autonomous agents
- Up-sampling rare events (e.g. fraud) to support more explainable models
- Open innovation and the mission behind the MOSTLY AI Prize
- Tools, SDKs, and open-source workflows for getting started with synthetic data
-  The MOSTLY AI Prize—a $100,000 global competition

Memorable Outtakes
“We need to move beyond thinking of real data as the gold standard. It’s often inaccessible, messy, biased—and by design, it's limited to how it was collected. Synthetic data lets us ask: what if our data was as inclusive as we needed it to be?”
“So much of synthetic data’s value is in unlocking what’s been locked away—allowing teams to safely build, test, and deploy where real data just isn’t viable.”
“It’s not just about boosting performance. Synthetic upsampling lets you use simpler, more explainable models—ones you can actually audit.”

References & Resources
- Alexandra Ebert – Chief AI & Data Democratization Officer, MOSTLY AI
 Industry profile: https://mostly.ai/team/alexandra-ebert
 LinkedIn: https://www.linkedin.com/in/alexandraebert/
-MOSTLY AI Prize – Global competition to advance privacy-preserving synthetic data
 Website: https://www.mostlyaiprize.com/
- MOSTLY AI GitHub & SDK – Open-source tools for structured synthetic data
 https://github.com/mostly-ai
-  https://github.com/mostly-ai/mostly-ai-sdk
- Synthetic Data Fairness Paper (ICLR)
 "Representative & Fair Synthetic Data"
 Paper link: https://arxiv.org/abs/2104.03007
- Synthetic Data Vault (SDV)  https://sdv.dev
- TVAE under the SDV umbrella: https://github.com/sdv-dev/SDV

Sponsored by
Agentic AI Summit 2025
 Join the premier virtual event for AI builders from July 15–30. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
 🔥 Use code podcast for 10% off any ticket.
 Register now: https://www.summit.ai/
ODSC West 2025 – The Leading AI Training Conference
 Join us in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
 🔥 Use code podcast for 10% off any ticket.
 Learn more: https://odsc.com/california

Beyond Real: The Case for Synthetic Data + How to Win $100K with Alexandra Ebert

In this episode of the ODSC AI Podcast, host Sheamus McGovern, founder of ODSC, sits down with Hugo Shi, Co-Founder and CTO of Saturn Cloud, a platform that gives data scientists and ML engineers the tools and flexibility they need to work the way they want. From his early days as a desk quant during the 2008 financial crisis to founding Saturn Cloud, Hugo brings a wealth of experience across finance, open source, and AI infrastructure.

This conversation dives deep into the realities of building AI infrastructure at scale, advocating for self-service tools for AI practitioners, managing cloud costs, and why flexibility—not control—is the foundation for productive data teams. It’s a must-listen for anyone working with machine learning infrastructure, whether you're a beginner navigating your first platform or a seasoned engineer scaling multi-cloud operations.

🌟 Key Topics Covered:
- Hugo’s career journey: From quant finance to co-founding Anaconda and then Saturn Cloud
- What working as a desk quant during the 2008 crisis taught him about speed, impatience, and iteration
- The pivotal role Anaconda played in democratizing Python data science
- Why Saturn Cloud was founded: common infra pain points across data teams
- How Saturn Cloud empowers teams through:
- - Interactive compute environments
- - Scheduled jobs
- - Long-running deployments
- The importance of flexibility vs. opinionated platforms 
- Why data scientists should not suffer in silence over infra pain
- Hidden cloud costs: compute, storage, and network—and how to manage them
- Differences between AI cloud providers (CoreWeave, Lambda Labs) and traditional hyperscalers (AWS, Azure, GCP)
- Scaling AI: lessons from working with massive clusters and thousands of parallel jobs
- Security best practices in ML platforms, including role-based access and cost attribution
- Why ML teams should collaborate across IT, product, and data engineering
- Hard-won lessons from real-world AI infrastructure scaling

💬 Memorable Outtakes:
On infrastructure friction and self-advocacy: “Data scientists, ML engineers, and AI engineers suffer in silence… They don’t perceive themselves as tech experts, so they think they have to accept infrastructure pain. They shouldn’t.”
On why Saturn Cloud avoids being too opinionated: “Notebooks are fine—but making them the only way to work? That’s a career trap. People should graduate to full IDEs and better practices.”
On scaling AI operations: “What can be done, will be done. If it’s possible, someone will try it. At scale, low-probability failures become inevitable.”

💻 References & Resources:
Hugo Shi:
- LinkedIn: https://www.linkedin.com/in/hugo-shi
- CTO & Co-Founder, Saturn Cloud: https://www.saturncloud.io/
Mentioned Companies and Tools:
- Saturn Cloud: https://www.saturncloud.io/
- Anaconda: https://www.anaconda.com/
- Phoenix Framework( For evaluation) https://github.com/phoenixframework/phoenix
- Prometheus (for resource monitoring): https://prometheus.io/
- CoreWeave: https://www.coreweave.com/
- Lambda Labs: https://lambdalabs.com/
- Neurelo (Neas): https://www.neurelo.com/
- RunPod: https://www.runpod.io/
- Cursor IDE: https://www.cursor.so/
- Streamlit: https://streamlit.io/
- Jupyter: https://jupyter.org/
- PyCharm: https://www.jetbrains.com/pycharm/

🤝 Sponsored by:
Agentic AI Summit 2025
Join the premier virtual event for AI builders from July 15–30. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
🔥 Use code podcast for 10% off any ticket.
Register now: https://www.summit.ai/
ODSC West 2025 – The Leading AI Training Conference
Attend in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
🔥 Use code podcast for 10% off any ticket.
Learn more: https://odsc.com/california

What No One Tells You About AI Infrastructure with Hugo Shi

AI Agents in Action: Memory, Messaging, and MCP with Michael Lanham

In this episode of the ODSC Podcast, Alex Landa, the head of content at ODSC, interviews Nick Walton, CEO and co-founder of Latitude and creator of the groundbreaking AI-powered storytelling game, AI Dungeon. Nick shares his journey from machine learning for self-driving cars to reimagining interactive fiction through generative models. They dive into the challenges and magic of building persistent, emergent game worlds with AI, and discuss how AI enables new modes of player agency and storytelling that go far beyond traditional game mechanics.
🔥 Key Topics Covered:
-The origin story of AI Dungeon and how GPT-2 sparked its creation
-How AI Dungeon enables open-ended, player-driven storytelling
-Technical and creative challenges in AI gaming: model curation, memory, and narrative tension
-What makes AI-generated characters compelling and emotionally resonant
-The upcoming game engine Latitude is building to power rich, persistent AI-driven worlds
-How AI can unlock new creative possibilities for indie and non-technical game creators
-Multiplayer and long-term visions for dynamic, socially evolving AI worlds
-Ethical considerations in using AI for gaming and storytelling
Memorable Outtakes:
💬 Nick Walton: “If every dwarf you meet is a gruff miner whose clan got eaten by goblins, it gets a little old after a while... you need higher variation.”
💬 Nick Walton: “You’re not just playing a game. You're building a story, and when loss happens, it becomes meaningful. That's the magic of AI Dungeon.”
💬 Nick Walton: “We’re building not just a game, but a living world—one where your choices shape everything, and the AI evolves with you.”
💬  Alex (Host): “It’s not about replacing anyone—it’s about creating something entirely new that couldn't exist without AI.”
🖥 References & Resources:
- AI Dungeon: https://aidungeon.com
-Latitude (Nick’s company): https://latitude.io
-Whispers from the Star (AI game mentioned): https://store.steampowered.com/app/3730100/Whispers_from_the_Star/
-Nick Walton’s X (Twitter) profile: https://twitter.com/nickwalton00
-Nick Walton’s LinkedIn: https://www.linkedin.com/in/waltonnick/
-Nick Walton's profile on Crunchbase: https://www.crunchbase.com/person/nick-walton
-AI Dungeon Dev Blog: https://latitude.io/blog


🌟 Sponsored by:
The Agentic AI Summit 2025
Attend the premier virtual event for AI builders from July 16–31. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
🔥 Use code podcast for 10% off any ticket.
Register now: https://www.summit.ai/
ODSC West 2025 – The Leading AI Training Conference
Join us in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
🔥 Use code podcast for 10% off any ticket.
Learn more: https://odsc.com/california


Creating Unique Narrative Experiences with Generative AI in Gaming with Nick Walton

In this episode of the ODSC Podcast, host Sheamus McGovern is joined by Veronika Durgin, Vice President of Data at Saks. Veronika brings over two decades of experience in data engineering, platform architecture, data modeling, and analytics. She joins us to expand on her ODSC presentation, The 10 Most Neglected Data Engineering Tasks, offering practical insights into often-overlooked practices that can make or break modern data systems. From defining what “done” really means to preparing for seasonality and building self-recovering pipelines, Veronika shares hard-won lessons, entertaining stories, and a deep well of practical advice.

💻 Key Topics Covered:
- Veronika’s unconventional career journey from biology to data leadership
- The “bridge strategy” in build vs. buy decisions and how it improves agility
- Why “definition of done” must include more than just working code
- The value of empathy: walking a mile in the business team’s shoes
- The dangers of ignoring business seasonality in data pipelines
- What makes a truly self-healing data pipeline
- Testing with production data vs. data in production
- Handling date/time data with precision and naming conventions
- The forgotten bucket of work: tech debt, improvements, and innovation
- The environmental impact of data systems and responsible engineering
- The rise of “vibe coding” and the challenges it poses for data engineering

💬 Memorable Outtakes:
- “If you don't know why you're working on this and who needs it, you shouldn't be working on it.” 
- “Everything we build today is tomorrow's legacy. Build things to be replaceable.”  
- “We automate, not because we’re lazy—but because we care about sleeping at night.” 

🖥 References & Resources:
- Veronika Durgin on LinkedIn: https://www.linkedin.com/in/vdurgin/
- Veronika Durgin on Medium: https://veronikadurgin.medium.com
- The Phoenix Project (pdf of book): https://shorturl.at/Ce5gA
- “Definition of Done”: https://medium.com/@durginv/definition-of-done-eea52c472cc3
- Tsedal Neeley Digital Mindset (book referenced): https://www.tsedal.com/book/the-digital-mindset/
- Stargate AI data center project (referenced during energy discussion): https://openai.com/index/announcing-the-stargate-project/

Sponsored by:
🔥 ODSC West 2025 – The Leading AI Training Conference
Join us in San Francisco from October 28th–30th for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
Use the code podcast for 10% off any ticket.
Learn more: https://odsc.com/california

The Most Neglected Tasks in Data Engineering with Veronika Durgin

In this episode of the ODSC AI Podcast, host Sheamus McGovern speaks with Ian Cairns, cofounder and CEO of Freeplay, a platform built to help teams evaluate, monitor, and iterate on LLM and agent-based systems in production. Ian brings a deep product background from Twitter, Gnip, and Mapbox, and offers an insider’s look into what it actually takes to make AI work beyond the prototype phase. The conversation centers on evaluation — widely regarded as one of the most difficult and underdeveloped aspects of deploying AI in 2025.

Key Topics Covered:
- The real-world AI maturity curve: from vibe prompting to production
- Offline vs. online evaluation: definitions, trade-offs, and tooling
- Why teams struggle post-deployment — and how to break through the “we don’t know what’s going wrong” phase
- Evaluation challenges with agents, memory, RAG, and tool use
- The role of observability, telemetry, and human-in-the-loop review
- Lessons learned from Freeplay customers, including Postscript
- The growing importance of domain experts in evaluation workflows
- Building multi-layer eval architectures for agent systems
- Voice agent challenges — like turn detection and latency
- Emerging roles like AI Evaluation Engineer and how orgs should staff for evaluation maturity

Memorable Outtakes:
- "The most mature teams start with their evals. They define what good looks like, then hill-climb toward that metric."
- "The breakthrough in quality comes from people getting close to the data. Sometimes, thousands of rows."

References & Resources:
- Freeplay website: https://www.freeplay.ai
- Deployed: The AI Product Podcast by Freeplay: https://open.spotify.com/show/6nZS3a7iYb2EzHcl78iNmi?si=de766e786a41461c&nd=1&dlsi=0cb3351f79644bfc
- Freeplay blog: https://www.freeplay.ai/blog
- Freeplay community newsletter: https://freeplay.ai/newsletter
- PipeCat (open-source voice agent toolkit): https://github.com/pipecat-ai/pipecat
- OpenTelemetry (agent observability framework): https://opentelemetry.io/
- Postscript (Freeplay customer case mentioned): https://www.postscript.io
- Colorado AI community meetups: https://www.boulderaibuilders.org/

Speaker Bio:
Ian Cairns is the CEO and co-founder of Freeplay. Previously, he served as Head of Product for Twitter’s Developer Platform, where he helped grow their enterprise data business from $40M to $400M ARR. He’s also worked at Gnip (acquired by Twitter), Mapbox, and in the Obama administration on open data initiatives. LinkedIn: https://www.linkedin.com/in/iancairns/

Sponsored by:
 🔥 ODSC West 2025 – The Leading AI Training Conference
 Join us in San Francisco from October 28th–30th for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
 Use the code podcast for 10% off any ticket.
 Learn more: https://odsc.com/california

The Hardest Problem in AI: Evaluation in 2025 with Ian Cairns

With Ai X Podcast, Open Data Science Conference (ODSC) brings its vast experience in building community and its knowledge of the data science and AI fields to the podcast platform. The interests and challenges of the data science community are wide ranging. To reflect this Ai X Podcast will offer a similarly wide range of content, from one-on-one interviews with leading experts, to career talks, to educational interviews, to profiles of AI Startup Founders. Join us every two weeks to discover what’s going on in the data science community.

Find more ODSC lightning interviews, webinars, live trainings, certifications, bootcamps here - https://aiplus.training/  
Don't miss out on this exciting opportunity to expand your knowledge and stay ahead of the curve.

Training and Deploying Open-Source LLMs with Dr. Jon Krohn

ODSC's Ai X Podcast

Related tracks