In this episode of the ODSC Podcast, we sit down with Sid Probstein, a seasoned enterprise technologist, 10-time CTO, and now CEO of SWIRL. Sid shares deep insights on the evolving role of AI search in the enterprise, the limitations of current AI architectures like RAG, and how federated search powered by large language models (LLMs) can offer a more scalable, secure, and efficient approach.
This wide-ranging conversation also touches on the future of agentic AI, the implications of zero-trust architecture, and why moving all your enterprise data into vector databases may not be the answer.
Key Topics Covered:
- Why many first-generation enterprise AI architectures fall short
- The common misconception that RAG requires vector databases
- How federated search is being reinvented with the help of LLMs
- The differences between RAG, AI search, and hybrid approaches
- Why data movement is a privacy, governance, and scaling bottleneck
- Using LLMs to evaluate, re-rank, and extract relevant results from existing search engines
- The impact of AI search on agentic workflows and enterprise automation
- Architectural considerations for building secure, scalable AI systems
- How open-source tools like Swirl can jumpstart your AI search strategy
- The future of applications and employment in an AI agent-driven world
Memorable Outtakes:
💬 "A lot of those first-generation architectures are just dead wrong."
 – Sid Probstein on the pitfalls of blindly applying RAG and vector database stacks in enterprise environments.
💬 "You do not need a vector database to do RAG. The idea that RAG is owned by the vector databases is absurd."
 – Sid debunks a common myth about RAG and data pipelines.
💬 "Federated search was niche—until LLMs changed everything."
 – Sid on how modern LLMs make sense of heterogeneous data and breathe new life into federated search.
References & Resources:
- Sid Probstein: https://www.linkedin.com/in/sidprobstein/
- Swirl (Open Source Project): https://github.com/swirlai/swirl-search
- Swirl Website: https://www.swirl.today
- Wired Article on COBOL Dates & Government Data (referenced in episode): https://www.wired.com/story/ai-small-business-administration-loans-errors/
- Estonia’s Digital Government Model (mentioned): https://e-estonia.com
- LangChain (for prompt engineering and agent workflows): https://www.langchain.com
- Open Source LLM Deployment with Ollama: https://ollama.com
- Deepseek R1:  https://github.com/deepseek-ai/DeepSeek-R1
Sponsored by:
This episode was sponsored by:
 🎤 ODSC East 2025 – The Leading AI Builders Conference – https://odsc.com/boston/
 Join us from May 13th to 15th in Boston for hands-on workshops, training sessions, and cutting-edge AI talks covering generative AI, LLMOps, and AI-driven automation.
🔥 Use the exclusive code ODSCPodcast for an additional 10% off any ticket type!

Rethinking RAG: Why AI Search Needs a New Architecture with Sid Probstein

In this episode of the ODSC Aix Podcast, host Sheamus McGovern speaks with Dr. Regina Barzilay, a Distinguished Professor of AI and Health at MIT, and one of the leading voices in applying artificial intelligence to real-world medical challenges. Dr. Barzilay is also the AI faculty lead at MIT’s Jameel Clinic, where her work spans early disease detection, personalized treatment, and AI-powered drug discovery.
A recipient of the MacArthur “Genius” Fellowship, the AAAI Squirrel AI Award, and a member of both the National Academy of Engineering and the National Academy of Medicine, Dr. Barzilay has built a career bridging state-of-the-art AI with the pressing needs of patients and clinicians.
Together, they explore why AI tools that can accurately predict cancer and other diseases are still rarely used in clinical practice—and what it will take to bridge the gap between research and real-world care.
Key Topics Covered
Why most diseases are diagnosed too late—and how AI can detect them before symptoms appear
The story behind MIRAI (for breast cancer) and SYBIL (for lung cancer) predictive models
Why powerful clinical AI tools aren’t widely adopted in hospitals and how the system can change
What it takes to get machine learning models into real-world care across countries and hospitals
The promise of generative AI in drug discovery and the discovery of new antibiotics
Challenges in detecting and treating neurodegenerative diseases like ALS
How AI could enable truly personalized medicine, predicting treatment side effects and responses
The role of AI copilots in clinical workflows and the future of clinician-AI collaboration
Why AI in healthcare must go beyond hype to deliver real, measurable value for patients



Memorable Outtakes
“The technology to save lives is here. We’re just not using it.” — Dr. Regina Barzilay on the gap between research and clinical adoption
“You can’t be an oncologist without using MRI or blood tests—but you can still practice medicine today without AI.” — Dr. Barzilay on structural resistance in healthcare systems
“Imagine giving some of your health data and getting back a real answer: what will happen to you. Not the population. You.” — Dr. Barzilay on personalized risk prediction and treatment modeling
References & Resources
Dr. Regina Barzilay  Academic Profile:https://www.csail.mit.edu/person/regina-barzilay LinkedIn:⁠ https://www.linkedin.com/in/reginabarzilay/⁠
Resources Mentioned in the Episode
MIRAI (AI model for breast cancer risk prediction): ⁠https://jclinic.mit.edu/mirai⁠
SYBIL (AI model for lung cancer risk prediction):  ⁠https://news.mit.edu/2023/ai-model-can-detect-future-lung-cancer-0120⁠
Halicin Antibiotic Discovery: 
MIT Jameel Clinic for Machine Learning in Health:⁠ https://jclinic.mit.edu/⁠
US Preventive Services Task Force (USPSTF):⁠ https://www.uspreventiveservicestaskforce.org/⁠
Professor Dina Katabi’s AI monitoring research (referenced): ⁠https://www.csail.mit.edu/person/dina-katabi
⁠

Sponsored by
Agentic AI Summit 2025 Join the premier virtual event for AI builders from July 15–30. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents. 🔥 Use code podcast for 10% off any ticket. Register now:⁠ https://www.summit.ai/⁠
ODSC West 2025 – The Leading AI Training Conference Attend in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation. 🔥 Use code podcast for 10% off any ticket. Learn more:⁠ https://odsc.com/california⁠

"AI Can Predict Disease—So Why Aren’t Doctors Using It?" with Regina Barzilay

This special episode captures compelling highlights from ODSC East 2025, held May 13–15. Join host Sheamus McGovern as he discusses transformative AI developments with leading experts, researchers, and industry practitioners. From the practical deployment of AI agents and the rising importance of synthetic data to ethical AI strategies and robust risk management practices, this episode offers diverse perspectives shaping the future of AI.

Guests and Key Topics Covered:
- Rob Bailey (CrewAI): Accelerating AI agent adoption in enterprises.
- Ivan Lee (Datasaur): The rise and effectiveness of specialized small language models.
- Tony Kipkemboi (CrewAI): Getting started with AI agents through practical, foundational skills.
- Samuel Colvin (Pydantic): Building robust, type-safe, and observable AI applications.
- Rajiv Shah (Contextual AI): Advanced evaluation methods for generative AI applications.
- Sinan Ozdemir (Author and entrepreneur) Sinan breaks down a comprehensive evaluation of AI agents and multi-agent systems, beyond benchmarks.
- Alexandra Ebert (MOSTLY AI): Ethical AI and synthetic data democratization.
- Kanchana Patlolla (Google Cloud): Overcoming common intelligent agent development and deployment challenges.
- Cal Al-Dhubaib & Lauren Burke-McCarthy (Further): AI risk management, red teaming, and sustainable data science adoption strategies.
- Dr. Andre Franca (Ergodic): Applying causal AI to improve decision-making processes.
- Noah Giansiracusa (Bentley University): Insights into social media algorithms and enhancing user autonomy.
- Allen Downey (PyMC Labs): Practical insights into traditional time series analysis and Bayesian modeling.

Memorable Outtakes:
- "Automation is no longer about wiring APIs together; it’s about launching self-directed agents that negotiate, learn, and act on our behalf." – Rob Bailey, CrewAI
- "Generative AI has turned traditional data science upside down. People forget that you still need rigorous evaluation to align models with the intended problem." – Rajiv Shah, Contextual AI
- "When introducing low-code and no-code platforms, you open the door to build quickly on shaky foundations." – Lauren Burke-McCarthy, Further

References & Resources:
- Rob Bailey (CrewAI): https://www.linkedin.com/in/robmbailey/
- Ivan Lee (Datasaur): https://www.linkedin.com/in/iylee/
- Tony Kipkemboi (CrewAI): https://www.linkedin.com/in/tonykipkemboi/
- Samuel Colvin (Pydantic): https://www.linkedin.com/in/samuel-colvin/
- Rajiv Shah (Contextual AI): https://www.linkedin.com/in/rajistics/
- Sinan Ozdemir (LoopGenius): https://www.linkedin.com/in/sinan-ozdemir/
- Alexandra Ebert (MOSTLY AI): https://www.linkedin.com/in/alexandraebert/
- Kanchana Patlolla (Google Cloud): https://www.linkedin.com/in/kanchanapatlolla/
- Cal Al-Dhubaib (Further): https://www.linkedin.com/in/dhubaib/
- Lauren Burke-McCarthy (Further): https://www.linkedin.com/in/lauren-e-burke/
- Dr. Andre Franca (Ergodic): https://www.linkedin.com/in/francaandre/
- Noah Giansiracusa, PhD (Bentley University): https://www.noahgian.com
- Allen Downey, PhD (PyMC Labs): https://www.linkedin.com/in/allendowney
- Humanity's Last Exam Benchmark:https://agi.safe.ai
- Pydantic: https://docs.pydantic.dev/
- MOSTLY AI Synthetic Data Competition: https://www.mostlyaiprize.com/
- MCP (Model Component Profiling):https://www.anthropic.com/news/model-context-protocol
- Datasaur: https://datasaur.ai/
- CrewAI: https://www.crewai.com/

Sponsored by:
- Agentic AI Summit 2025: Join the premier virtual event for AI builders from July 16–31. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
🔥 Use code podcast for 10% off any ticket.
Register now: https://www.summit.ai/

- ODSC West 2025 – The Leading AI Training Conference
Attend in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
🔥 Use code podcast for 10% off any ticket.
Learn more: https://odsc.com/california

ODSC East 2025 Minisodes

In this episode of the ODSC AIx Podcast, host Sheamus McGovern speaks with Alexandra Ebert, Chief AI and Data Democratization Officer at MOSTLY AI, and one of the foremost voices on privacy-preserving synthetic data and responsible AI. With a diverse background spanning AI ethics, data access policy, and generative AI regulation, Alexandra brings clarity to how synthetic data is not just a privacy tool but a lever for innovation, fairness, and scalable AI adoption.
The discussion explores what synthetic data is (and isn’t), its core advantages and limitations, its role in addressing fairness and access challenges in data-driven organizations, and how practitioners can actively shape better downstream model performance. The episode also dives into the MOSTLY AI Prize—a $100,000 global competition to advance privacy-safe, high-utility synthetic data generation.

Key Topics Covered:
- The different types and use cases of synthetic data (privacy-preserving, simulation-based, creative)
- How synthetic data helps solve the “data access paradox” in regulated industries
- Key advantages and limitations of synthetic data vs. real-world and legacy anonymized data
- Privacy mechanisms: Outlier suppression, statistical mimicry, empirical differential privacy
Real-world use cases in healthcare, finance, telco, and simulation environments
- Fairness-aware synthetic data generation using statistical parity constraints
- Imputing missing data with synthetic distributions
- Agentic AI and the role of synthetic data in enabling secure access layers for autonomous agents
- Up-sampling rare events (e.g. fraud) to support more explainable models
- Open innovation and the mission behind the MOSTLY AI Prize
- Tools, SDKs, and open-source workflows for getting started with synthetic data
-  The MOSTLY AI Prize—a $100,000 global competition

Memorable Outtakes
“We need to move beyond thinking of real data as the gold standard. It’s often inaccessible, messy, biased—and by design, it's limited to how it was collected. Synthetic data lets us ask: what if our data was as inclusive as we needed it to be?”
“So much of synthetic data’s value is in unlocking what’s been locked away—allowing teams to safely build, test, and deploy where real data just isn’t viable.”
“It’s not just about boosting performance. Synthetic upsampling lets you use simpler, more explainable models—ones you can actually audit.”

References & Resources
- Alexandra Ebert – Chief AI & Data Democratization Officer, MOSTLY AI
 Industry profile: https://mostly.ai/team/alexandra-ebert
 LinkedIn: https://www.linkedin.com/in/alexandraebert/
-MOSTLY AI Prize – Global competition to advance privacy-preserving synthetic data
 Website: https://www.mostlyaiprize.com/
- MOSTLY AI GitHub & SDK – Open-source tools for structured synthetic data
 https://github.com/mostly-ai
-  https://github.com/mostly-ai/mostly-ai-sdk
- Synthetic Data Fairness Paper (ICLR)
 "Representative & Fair Synthetic Data"
 Paper link: https://arxiv.org/abs/2104.03007
- Synthetic Data Vault (SDV)  https://sdv.dev
- TVAE under the SDV umbrella: https://github.com/sdv-dev/SDV

Sponsored by
Agentic AI Summit 2025
 Join the premier virtual event for AI builders from July 15–30. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
 🔥 Use code podcast for 10% off any ticket.
 Register now: https://www.summit.ai/
ODSC West 2025 – The Leading AI Training Conference
 Join us in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
 🔥 Use code podcast for 10% off any ticket.
 Learn more: https://odsc.com/california

Beyond Real: The Case for Synthetic Data + How to Win $100K with Alexandra Ebert

In this episode of the ODSC AI Podcast, host Sheamus McGovern, founder of ODSC, sits down with Hugo Shi, Co-Founder and CTO of Saturn Cloud, a platform that gives data scientists and ML engineers the tools and flexibility they need to work the way they want. From his early days as a desk quant during the 2008 financial crisis to founding Saturn Cloud, Hugo brings a wealth of experience across finance, open source, and AI infrastructure.

This conversation dives deep into the realities of building AI infrastructure at scale, advocating for self-service tools for AI practitioners, managing cloud costs, and why flexibility—not control—is the foundation for productive data teams. It’s a must-listen for anyone working with machine learning infrastructure, whether you're a beginner navigating your first platform or a seasoned engineer scaling multi-cloud operations.

🌟 Key Topics Covered:
- Hugo’s career journey: From quant finance to co-founding Anaconda and then Saturn Cloud
- What working as a desk quant during the 2008 crisis taught him about speed, impatience, and iteration
- The pivotal role Anaconda played in democratizing Python data science
- Why Saturn Cloud was founded: common infra pain points across data teams
- How Saturn Cloud empowers teams through:
- - Interactive compute environments
- - Scheduled jobs
- - Long-running deployments
- The importance of flexibility vs. opinionated platforms 
- Why data scientists should not suffer in silence over infra pain
- Hidden cloud costs: compute, storage, and network—and how to manage them
- Differences between AI cloud providers (CoreWeave, Lambda Labs) and traditional hyperscalers (AWS, Azure, GCP)
- Scaling AI: lessons from working with massive clusters and thousands of parallel jobs
- Security best practices in ML platforms, including role-based access and cost attribution
- Why ML teams should collaborate across IT, product, and data engineering
- Hard-won lessons from real-world AI infrastructure scaling

💬 Memorable Outtakes:
On infrastructure friction and self-advocacy: “Data scientists, ML engineers, and AI engineers suffer in silence… They don’t perceive themselves as tech experts, so they think they have to accept infrastructure pain. They shouldn’t.”
On why Saturn Cloud avoids being too opinionated: “Notebooks are fine—but making them the only way to work? That’s a career trap. People should graduate to full IDEs and better practices.”
On scaling AI operations: “What can be done, will be done. If it’s possible, someone will try it. At scale, low-probability failures become inevitable.”

💻 References & Resources:
Hugo Shi:
- LinkedIn: https://www.linkedin.com/in/hugo-shi
- CTO & Co-Founder, Saturn Cloud: https://www.saturncloud.io/
Mentioned Companies and Tools:
- Saturn Cloud: https://www.saturncloud.io/
- Anaconda: https://www.anaconda.com/
- Phoenix Framework( For evaluation) https://github.com/phoenixframework/phoenix
- Prometheus (for resource monitoring): https://prometheus.io/
- CoreWeave: https://www.coreweave.com/
- Lambda Labs: https://lambdalabs.com/
- Neurelo (Neas): https://www.neurelo.com/
- RunPod: https://www.runpod.io/
- Cursor IDE: https://www.cursor.so/
- Streamlit: https://streamlit.io/
- Jupyter: https://jupyter.org/
- PyCharm: https://www.jetbrains.com/pycharm/

🤝 Sponsored by:
Agentic AI Summit 2025
Join the premier virtual event for AI builders from July 15–30. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
🔥 Use code podcast for 10% off any ticket.
Register now: https://www.summit.ai/
ODSC West 2025 – The Leading AI Training Conference
Attend in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
🔥 Use code podcast for 10% off any ticket.
Learn more: https://odsc.com/california

What No One Tells You About AI Infrastructure with Hugo Shi

AI Agents in Action: Memory, Messaging, and MCP with Michael Lanham

In this episode of the ODSC Podcast, Alex Landa, the head of content at ODSC, interviews Nick Walton, CEO and co-founder of Latitude and creator of the groundbreaking AI-powered storytelling game, AI Dungeon. Nick shares his journey from machine learning for self-driving cars to reimagining interactive fiction through generative models. They dive into the challenges and magic of building persistent, emergent game worlds with AI, and discuss how AI enables new modes of player agency and storytelling that go far beyond traditional game mechanics.
🔥 Key Topics Covered:
-The origin story of AI Dungeon and how GPT-2 sparked its creation
-How AI Dungeon enables open-ended, player-driven storytelling
-Technical and creative challenges in AI gaming: model curation, memory, and narrative tension
-What makes AI-generated characters compelling and emotionally resonant
-The upcoming game engine Latitude is building to power rich, persistent AI-driven worlds
-How AI can unlock new creative possibilities for indie and non-technical game creators
-Multiplayer and long-term visions for dynamic, socially evolving AI worlds
-Ethical considerations in using AI for gaming and storytelling
Memorable Outtakes:
💬 Nick Walton: “If every dwarf you meet is a gruff miner whose clan got eaten by goblins, it gets a little old after a while... you need higher variation.”
💬 Nick Walton: “You’re not just playing a game. You're building a story, and when loss happens, it becomes meaningful. That's the magic of AI Dungeon.”
💬 Nick Walton: “We’re building not just a game, but a living world—one where your choices shape everything, and the AI evolves with you.”
💬  Alex (Host): “It’s not about replacing anyone—it’s about creating something entirely new that couldn't exist without AI.”
🖥 References & Resources:
- AI Dungeon: https://aidungeon.com
-Latitude (Nick’s company): https://latitude.io
-Whispers from the Star (AI game mentioned): https://store.steampowered.com/app/3730100/Whispers_from_the_Star/
-Nick Walton’s X (Twitter) profile: https://twitter.com/nickwalton00
-Nick Walton’s LinkedIn: https://www.linkedin.com/in/waltonnick/
-Nick Walton's profile on Crunchbase: https://www.crunchbase.com/person/nick-walton
-AI Dungeon Dev Blog: https://latitude.io/blog


🌟 Sponsored by:
The Agentic AI Summit 2025
Attend the premier virtual event for AI builders from July 16–31. Gain hands-on skills in designing, deploying, and scaling autonomous AI agents.
🔥 Use code podcast for 10% off any ticket.
Register now: https://www.summit.ai/
ODSC West 2025 – The Leading AI Training Conference
Join us in San Francisco from October 28–30 for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
🔥 Use code podcast for 10% off any ticket.
Learn more: https://odsc.com/california


Creating Unique Narrative Experiences with Generative AI in Gaming with Nick Walton

In this episode of the ODSC Podcast, host Sheamus McGovern is joined by Veronika Durgin, Vice President of Data at Saks. Veronika brings over two decades of experience in data engineering, platform architecture, data modeling, and analytics. She joins us to expand on her ODSC presentation, The 10 Most Neglected Data Engineering Tasks, offering practical insights into often-overlooked practices that can make or break modern data systems. From defining what “done” really means to preparing for seasonality and building self-recovering pipelines, Veronika shares hard-won lessons, entertaining stories, and a deep well of practical advice.

💻 Key Topics Covered:
- Veronika’s unconventional career journey from biology to data leadership
- The “bridge strategy” in build vs. buy decisions and how it improves agility
- Why “definition of done” must include more than just working code
- The value of empathy: walking a mile in the business team’s shoes
- The dangers of ignoring business seasonality in data pipelines
- What makes a truly self-healing data pipeline
- Testing with production data vs. data in production
- Handling date/time data with precision and naming conventions
- The forgotten bucket of work: tech debt, improvements, and innovation
- The environmental impact of data systems and responsible engineering
- The rise of “vibe coding” and the challenges it poses for data engineering

💬 Memorable Outtakes:
- “If you don't know why you're working on this and who needs it, you shouldn't be working on it.” 
- “Everything we build today is tomorrow's legacy. Build things to be replaceable.”  
- “We automate, not because we’re lazy—but because we care about sleeping at night.” 

🖥 References & Resources:
- Veronika Durgin on LinkedIn: https://www.linkedin.com/in/vdurgin/
- Veronika Durgin on Medium: https://veronikadurgin.medium.com
- The Phoenix Project (pdf of book): https://shorturl.at/Ce5gA
- “Definition of Done”: https://medium.com/@durginv/definition-of-done-eea52c472cc3
- Tsedal Neeley Digital Mindset (book referenced): https://www.tsedal.com/book/the-digital-mindset/
- Stargate AI data center project (referenced during energy discussion): https://openai.com/index/announcing-the-stargate-project/

Sponsored by:
🔥 ODSC West 2025 – The Leading AI Training Conference
Join us in San Francisco from October 28th–30th for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
Use the code podcast for 10% off any ticket.
Learn more: https://odsc.com/california

The Most Neglected Tasks in Data Engineering with Veronika Durgin

In this episode of the ODSC AI Podcast, host Sheamus McGovern speaks with Ian Cairns, cofounder and CEO of Freeplay, a platform built to help teams evaluate, monitor, and iterate on LLM and agent-based systems in production. Ian brings a deep product background from Twitter, Gnip, and Mapbox, and offers an insider’s look into what it actually takes to make AI work beyond the prototype phase. The conversation centers on evaluation — widely regarded as one of the most difficult and underdeveloped aspects of deploying AI in 2025.

Key Topics Covered:
- The real-world AI maturity curve: from vibe prompting to production
- Offline vs. online evaluation: definitions, trade-offs, and tooling
- Why teams struggle post-deployment — and how to break through the “we don’t know what’s going wrong” phase
- Evaluation challenges with agents, memory, RAG, and tool use
- The role of observability, telemetry, and human-in-the-loop review
- Lessons learned from Freeplay customers, including Postscript
- The growing importance of domain experts in evaluation workflows
- Building multi-layer eval architectures for agent systems
- Voice agent challenges — like turn detection and latency
- Emerging roles like AI Evaluation Engineer and how orgs should staff for evaluation maturity

Memorable Outtakes:
- "The most mature teams start with their evals. They define what good looks like, then hill-climb toward that metric."
- "The breakthrough in quality comes from people getting close to the data. Sometimes, thousands of rows."

References & Resources:
- Freeplay website: https://www.freeplay.ai
- Deployed: The AI Product Podcast by Freeplay: https://open.spotify.com/show/6nZS3a7iYb2EzHcl78iNmi?si=de766e786a41461c&nd=1&dlsi=0cb3351f79644bfc
- Freeplay blog: https://www.freeplay.ai/blog
- Freeplay community newsletter: https://freeplay.ai/newsletter
- PipeCat (open-source voice agent toolkit): https://github.com/pipecat-ai/pipecat
- OpenTelemetry (agent observability framework): https://opentelemetry.io/
- Postscript (Freeplay customer case mentioned): https://www.postscript.io
- Colorado AI community meetups: https://www.boulderaibuilders.org/

Speaker Bio:
Ian Cairns is the CEO and co-founder of Freeplay. Previously, he served as Head of Product for Twitter’s Developer Platform, where he helped grow their enterprise data business from $40M to $400M ARR. He’s also worked at Gnip (acquired by Twitter), Mapbox, and in the Obama administration on open data initiatives. LinkedIn: https://www.linkedin.com/in/iancairns/

Sponsored by:
 🔥 ODSC West 2025 – The Leading AI Training Conference
 Join us in San Francisco from October 28th–30th for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
 Use the code podcast for 10% off any ticket.
 Learn more: https://odsc.com/california

The Hardest Problem in AI: Evaluation in 2025 with Ian Cairns

In this episode of ODSC’s AiX Podcast, Sheamus McGovern sits down with Dan Huss, the Founder & CEO of Gravity AI, to unpack how AI is changing product management. The conversation explores how AI is reshaping product management, from evaluating AI’s true value, avoiding AI washing, and differentiating between AI as a feature versus AI as a core product, to Dan’s concept of the Minimum Viable Experiment (MVE) and Minimum Viable AI (MVAI ) as an alternative to MVP. They also dive into the evolving skill sets product managers need in the AI era, cross-functional collaboration with data science teams, and responsible AI implementation.

Key Topics Covered:
- MVAI (Minimum Viable AI)  and the future of product management.
- Why MVP gets bloated in the enterprise and why MVE (Minimum Viable Experiment) prioritizes learning over shipping.
- A product lens for AI decisions using desirability, feasibility, viability — including consequences when models are wrong.
- AI-as-a-feature vs. AI-as-a-product — what changes in data strategy and UI (e.g., conversational interfaces).
- Where AI impacts the PM workflow: user stories, specs, research synthesis, and “vibe coding” for rapid prototypes.
- Collaboration playbook for PMs ↔ Data Science / AI Engineering on hallucinations, accuracy, monitoring, and risk.
- Practical MLOps: catalogs, deployment, observability, and compliance (e.g., EU AI Act implications).
- Avoiding AI washing: How to find low‑risk, high‑learning internal use cases and build toward differentiation

Memorable Outtakes:
- “Ditch MVP. Let’s call it an MVE—Minimum Viable Experiment. Put the emphasis on learning, not building a ‘version one’ that gets bloated.”
- “Product management is fundamentally a communications role. Anywhere there’s a language task, AI will become a collaborator.”
References & Resources:

Guest & Company
- - Dan Huss (Founder & CEO, Gravity AI) — LinkedIn: https://www.linkedin.com/in/danielhuss/
Gravity AI: https://www.gravity-ai.com/
Concepts, Tools & Mentions from the Episode
- Minimum Viable Product: https://en.wikipedia.org/wiki/Minimum_viable_product
- The Lean Startup https://en.wikipedia.org/wiki/The_Lean_Startup
- AI Snake Oil (book/site): https://www.aisnakeoil.com
- Podcast Episode - The AI Superintelligence Myth with Arvind Narayanan: https://creators.spotify.com/pod/profile/ai-x-podcast/episodes/The-AI-Superintelligence-Myth-with-Arvind-Narayanan-e32it16
- EU AI Act (overview resource): https://artificialintelligenceact.eu/
- Notion: https://www.notion.so/ 
- Figma: https://www.figma.com/

Sponsored by:
 🔥 ODSC West 2025 – The Leading AI Training Conference
Join us in San Francisco from October 28th–30th for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
Use the code podcast for 10% off any ticket.
Learn more: https://odsc.com/california

Minimum Viable AI: Redefining How We Build Products with Dan Huss

In this special episode of the ODSC AI+ Podcast, host Sheamus McGovern dives into the real-world impact of GPT-5—from routing and hallucination issues to cost savings and open-weight models.

Joining him are two expert guests:
- Ivan Lee: Founder and CEO of Datasaur, who helps enterprises build private LLM stacks and has deep experience evaluating model upgrades.
-Nir Gazit: Co-founder and CEO of Traceloop, and co-creator of the OpenTelemetry Generative AI SIG, who brings insight into model routing, evaluation strategies, and observability tooling.
Together, they unpack what GPT-5 actually changed—and what teams should do next.

Key Topics Covered:
-Why GPT-5’s biggest shift is routing, not reasoning
-What casual vs. power users gained (or lost) with the rollout
-Hallucination benchmarks vs. real-world results
-Evaluation strategies using open-source tools like Phoenix and LangChain
-OpenAI’s OSS model release and its enterprise implications
-Why developers worry about black-box routing and the lack of traceability
-How to migrate safely: pinning snapshots, running evals, shadow testing
-Whether GPT-5 gets us closer to AGI, or just better infrastructure
-What to expect from agent workflows, tool selection, and model specialization

Memorable Outtakes:
- Ivan Lee: “GPT-5 is an upgrade for 98% of users—but for the power users, the loss of model choice felt like control was taken away.”
- Nir Gazit: “Of course, every new model crushes it on benchmarks—they’re optimizing for the benchmarks. That doesn’t mean it works for your use case.”
- Ivan Lee: “OpenAI’s OSS release might be the bigger story than GPT-5. Suddenly, enterprises are back at the table.”

References & Resources:
Guests
- Ivan Lee – CEO of Datasaur
Website: https://www.datasaur.ai
LinkedIn: https://www.linkedin.com/in/iylee/
- Nir Gazit – CEO of Traceloop
Website: https://www.traceloop.com
Blog: https://www.traceloop.com/blog
LinkedIn: https://www.linkedin.com/in/nirga/

Resources Mentioned
- OpenAI GPT-5  https://openai.com/gpt-5
- OpenTelemetry Project: https://opentelemetry.io
- Traceloop OpenLLMetry: https://www.traceloop.com/openllmetry
- Phoenix (Arize AI open-source evals): https://github.com/Arize-ai/phoenix
- LangChain Evals: https://python.langchain.com/api_reference/langchain/evaluation.html
- GPT-OSS Open Weight Models by OpenAI: https://platform.openai.com/docs/models/gpt-oss
- Claude + Model Context Protocol (Anthropic): https://docs.anthropic.com/en/docs/tool-use
- ARC-AGI Leaderboard: https://arcprize.org/leaderboard

Sponsored by:
🔥 ODSC AI West 2025 – The Leading AI Training Conference
Join us in San Francisco from October 28th–30th for expert-led sessions on generative AI, LLMOps, and AI-driven automation.
Use the code podcast for 10% off any ticket.
Learn more: https://odsc.ai 


GPT-5 Unboxed: What Changed, What Broke, and What’s Next with Ivan Lee and Nir Gazit

With Ai X Podcast, Open Data Science Conference (ODSC) brings its vast experience in building community and its knowledge of the data science and AI fields to the podcast platform. The interests and challenges of the data science community are wide ranging. To reflect this Ai X Podcast will offer a similarly wide range of content, from one-on-one interviews with leading experts, to career talks, to educational interviews, to profiles of AI Startup Founders. Join us every two weeks to discover what’s going on in the data science community.

Find more ODSC lightning interviews, webinars, live trainings, certifications, bootcamps here - https://aiplus.training/  
Don't miss out on this exciting opportunity to expand your knowledge and stay ahead of the curve.

Rethinking RAG: Why AI Search Needs a New Architecture with Sid Probstein

ODSC's Ai X Podcast