In this episode of the ODSC Podcast, we sit down with Sid Probstein, a seasoned enterprise technologist, 10-time CTO, and now CEO of SWIRL. Sid shares deep insights on the evolving role of AI search in the enterprise, the limitations of current AI architectures like RAG, and how federated search powered by large language models (LLMs) can offer a more scalable, secure, and efficient approach.
This wide-ranging conversation also touches on the future of agentic AI, the implications of zero-trust architecture, and why moving all your enterprise data into vector databases may not be the answer.
Key Topics Covered:
- Why many first-generation enterprise AI architectures fall short
- The common misconception that RAG requires vector databases
- How federated search is being reinvented with the help of LLMs
- The differences between RAG, AI search, and hybrid approaches
- Why data movement is a privacy, governance, and scaling bottleneck
- Using LLMs to evaluate, re-rank, and extract relevant results from existing search engines
- The impact of AI search on agentic workflows and enterprise automation
- Architectural considerations for building secure, scalable AI systems
- How open-source tools like Swirl can jumpstart your AI search strategy
- The future of applications and employment in an AI agent-driven world
Memorable Outtakes:
💬 "A lot of those first-generation architectures are just dead wrong."
 – Sid Probstein on the pitfalls of blindly applying RAG and vector database stacks in enterprise environments.
💬 "You do not need a vector database to do RAG. The idea that RAG is owned by the vector databases is absurd."
 – Sid debunks a common myth about RAG and data pipelines.
💬 "Federated search was niche—until LLMs changed everything."
 – Sid on how modern LLMs make sense of heterogeneous data and breathe new life into federated search.
References & Resources:
- Sid Probstein: https://www.linkedin.com/in/sidprobstein/
- Swirl (Open Source Project): https://github.com/swirlai/swirl-search
- Swirl Website: https://www.swirl.today
- Wired Article on COBOL Dates & Government Data (referenced in episode): https://www.wired.com/story/ai-small-business-administration-loans-errors/
- Estonia’s Digital Government Model (mentioned): https://e-estonia.com
- LangChain (for prompt engineering and agent workflows): https://www.langchain.com
- Open Source LLM Deployment with Ollama: https://ollama.com
- Deepseek R1:  https://github.com/deepseek-ai/DeepSeek-R1
Sponsored by:
This episode was sponsored by:
 🎤 ODSC East 2025 – The Leading AI Builders Conference – https://odsc.com/boston/
 Join us from May 13th to 15th in Boston for hands-on workshops, training sessions, and cutting-edge AI talks covering generative AI, LLMOps, and AI-driven automation.
🔥 Use the exclusive code ODSCPodcast for an additional 10% off any ticket type!

Rethinking RAG: Why AI Search Needs a New Architecture with Sid Probstein

With Ai X Podcast, Open Data Science Conference (ODSC) brings its vast experience in building community and its knowledge of the data science and AI fields to the podcast platform. The interests and challenges of the data science community are wide ranging. To reflect this Ai X Podcast will offer a similarly wide range of content, from one-on-one interviews with leading experts, to career talks, to educational interviews, to profiles of AI Startup Founders. Join us every two weeks to discover what’s going on in the data science community.

Find more ODSC lightning interviews, webinars, live trainings, certifications, bootcamps here - https://aiplus.training/  
Don't miss out on this exciting opportunity to expand your knowledge and stay ahead of the curve.

Rethinking RAG: Why AI Search Needs a New Architecture with Sid Probstein

ODSC's Ai X Podcast