Reinforcement learning through human feedback (RLHF) has come a long way. In this episode, research scientist Nathan Lambert talks to Jon Krohn about the technique’s origins of the technique. He also walks through other ways to fine-tune LLMs, and how he believes generative AI might democratize education.

This episode is brought to you by AWS Inferentia (https://go.aws/3zWS0au) and AWS Trainium (https://go.aws/3ycV6K0), and Crawlbase (https://crawlbase.com), the ultimate data crawling platform. Interested in sponsoring a SuperDataScience Podcast episode? Email natalie@superdatascience.com for sponsorship information.

In this episode you will learn:
• Why it is important that AI is open [03:13]
• The efficacy and scalability of direct preference optimization [07:32]
• Robotics and LLMs [14:32]
• The challenges to aligning reward models with human preferences [23:00]
• How to make sure AI’s decision making on preferences reflect desirable behavior [28:52]
• Why Nathan believes AI is closer to alchemy than science [37:38]

Additional materials: www.superdatascience.com/791

The Super Data Science Podcast: Machine Learning | AI | Success

791: Reinforcement Learning from Human Feedback (RLHF), with Dr. Nathan Lambert

In this podcast, Roland and Anthony discuss the use of simulation of both musical instruments and robots. They explore how software and sampling techniques allow musicians to replicate the sounds of real instruments and to design better pianos before manufacturing. They discuss how robot simulations allow testing code safely in virtual environments, avoiding costly or dangerous real-world consequences. The conversation extends to frameworks like Gazebo, Unity, and reinforcement learning platforms, and also how simulations cannot always bridge the gap between training in virtual spaces and real-world execution.

Read a transcript of this interview: https://bit.ly/4ff74TB

Subscribe to the Software Architects’ Newsletter for your monthly guide to the essential news and experience from industry peers on emerging patterns and technologies:
https://www.infoq.com/software-architects-newsletter 

Upcoming Events:

QCon San Francisco (November 18-22, 2024)
Get practical inspiration and best practices on emerging software trends directly from senior software developers at early adopter companies.
https://qconsf.com/ 

QCon London (April 7-9, 2025)
Discover new ideas and insights from senior practitioners driving change and innovation in software development.
https://qconlondon.com/ 

Save the date: InfoQ Dev Summit Boston (June 9-10, 2025)
Actionable insights on today’s critical dev priorities.


The InfoQ Podcasts: 
Weekly inspiration to drive innovation and build great teams from senior software leaders. Listen to all our podcasts and read interview transcripts:

- The InfoQ Podcast https://www.infoq.com/podcasts/ 
- Engineering Culture Podcast by InfoQ https://www.infoq.com/podcasts/#engineering_culture 
- Generally AI: https://www.infoq.com/generally-ai-podcast/ 

Follow InfoQ:
- Mastodon: https://techhub.social/@infoq 
- Twitter: twitter.com/InfoQ
- LinkedIn: www.linkedin.com/company/infoq
- Facebook: bit.ly/2jmlyG8
- Instagram: @infoqdotcom
- Youtube: www.youtube.com/infoq

Write for InfoQ:
Learn and share the changes and innovations in professional software development.
- Join a community of experts.
- Increase your visibility.
- Grow your career.
https://www.infoq.com/write-for-infoq

Transcript

Generally AI - Season 2 - Episode 5: Do Robots Dream of Electric Pianos?

Generated Podcast in NotebookLM based on paper https://arxiv.org/abs/2405.12399 "Diffusion for World Modeling: Visual Details Matter in Atari" (20 May 2024)

Diffusion for World Modelling in Reinforcement Learning V1

The NotebookLM generation as review of the paper "Diffusion for World Modelling - Visual Details Matter in Atari" (2024) https://arxiv.org/abs/2405.12399 with tight connection to other papers in Reinforcement Learning with World Modeling.

Paper review - Diffusion for World Modelling - Visual Details Matter in Atari

Paper 1901.03729 - Automated Rationale Generation_ A Technique for Explainable AI and its Effects on Human Perceptions

Good morning! It's January 20th, 2025, and welcome to AI Tech News Today. Here are the latest headlines in the world of Artificial Intelligence!

Man builds nuclear fusion reactor in kitchen using Claude AI. This might sound like science fiction, but user HudZah took on this unconventional project by successfully building a neutron-producing nuclear fusor right in his kitchen. Streamed over a thirty-six-hour live session, he utilized Claude A-I as his co-pilot for every step, from obtaining components to piecing it all together safely. With parts like a thirty thousand-volt electrostatic precipitator and a highly efficient vacuum system, HudZah turned dreams of fusion into reality. Just don’t try this one at home!

OpenAI’s "Ph.D.-caliber super-agents" set to debut, but Altman warns against hype. OpenAI is preparing to unveil advanced AI systems called “Ph.D.-level super-agents,” designed to tackle complex challenges with expert-level insight. However, CEO Sam Altman advised everyone to “chill and cut expectations a hundred times.” While potential exists for these powerful agents, Altman cautions against assuming we’re just a stone’s throw from true artificial general intelligence. Expect incremental improvements instead of miracles!

Runway’s new AI image generator is here, and it looks fittingly cinematic. Runway has released "Frames," a text-to-image generator focused on creating high-quality images for professional use. With the ability to style images across nineteen presets—from “Vivid Warm” to “Terracotta”—it’s aimed at creative professionals who want consistent visuals. The model even allows for quick transformations from still images to videos. Pretty cool, right?

Humanoid robot masters the waltz by mimicking human movement. Researchers at UC San Diego developed ExBody2, an AI system that enables humanoid robots to learn and replicate human movements smoothly, including the classic waltz. By using motion capture data and reinforcement learning, the robots can adapt to real-world scenarios, opening the door for greater coordination and flexibility in humanoid robotics. Who knew dancing robots could be a thing?

That’s all for today’s AI news! Stay tuned and continue exploring the dynamic world of Artificial Intelligence with us. Keep those inquisitive minds engaged, and we will see you next time!

Kitchen Fusion: DIY Enthusiast Constructs Nuclear Reactor with Claude AI's Help

Hello and welcome back to AI Tech News Today! It’s February third, twenty twenty-five, and we have a critical update and some fascinating news in the world of Artificial Intelligence. Let’s dive right into it.

deepseek exposed user data, chat histories in open database. Researchers from cloud security firm Wiz discovered that DeepSeek had an unprotected database that left sensitive user information, such as chat histories, API authentication keys, and system logs, in plain view for anyone to access. The database was publicly accessible and required no authentication, raising serious concerns about data security and privacy.

berkeley researchers replicate deepseek AI for just thirty dollars. In response to the vulnerabilities identified with DeepSeek, researchers from the University of California, Berkeley, have developed a new model called TinyZero. This smaller-scale alternative utilizes reinforcement learning and is available on GitHub for public experimentation—all for the low price of thirty dollars.

openAI o3-mini: faster, smarter AI for STEM, now free for ChatGPT users. This latest model is targeted at science, technology, engineering, and mathematics fields, providing enhanced capabilities while lowering costs and reducing latency. Brought to you free of charge for ChatGPT users, it builds on the advancements of its predecessor while pushing the boundaries for small-scale AI applications.

openAI unveils ChatGPT deep research agent for in-depth AI analysis. The new Deep Research agent is designed for professionals in finance, science, engineering, and policy-making, offering a more analytical approach than traditional chat responses. It supports users researching significant purchases, such as cars and appliances, providing detailed analysis to facilitate informed decisions.

stanford’s virtual lab uses AI agents to advance scientific research. Stanford researchers are utilizing specialized AI agents to simulate collaborations among chemists, biologists, and machine learning experts. This innovative approach has the potential to accelerate breakthroughs across various fields, including medicine, climate science, and engineering.

That wraps up today’s AI Tech News! As the development of artificial intelligence continues to evolve rapidly, it’s crucial to stay informed about the implications for privacy, security, and technological advancements. Keep tuning in and remember: the future is bright for AI, so let’s keep our eyes on the horizon!

DeepSeek Data Breach Sparks Development of Affordable AI Alternatives and Advanced Research Tools

🎙️ Welcome to "AI with Shaily," your trusted source for the latest and greatest in artificial intelligence breakthroughs! I'm Shailendra Kumar, and today we're exploring some thrilling advancements in reinforcement learning (RL) that are making AI not only smarter and faster but also more environmentally friendly. 🌱🤖

Imagine training a robot to perform tasks without guzzling massive amounts of computing power. That’s the goal of cutting-edge RL research. One standout innovation involves using randomized neural networks within a new actor-critic algorithm. While it sounds complex, think of it as injecting a bit of creative randomness into AI’s learning process. This clever twist speeds up training and dramatically cuts down on computational effort. As someone who's spent countless hours debugging AI models, I find this approach like discovering a shortcut through a maze—elegant, efficient, and even a bit fun! 🧩⚡

Another fascinating breakthrough comes from applying reinforcement learning after a language model has already been trained. This technique helps AI speak more concisely without sacrificing accuracy. Imagine your AI assistant delivering sharp, to-the-point responses, saving both time and energy. It’s like trimming the fluff from a story to reveal only the most important parts. I remember asking an AI chatbot to summarize a book once, and it spun a 10-minute epic saga! This new method could have saved me many coffee breaks. ☕✂️📚

Why does this matter beyond the tech? Reinforcement learning is no longer just a lab experiment—it’s driving real-world applications. From autonomous vehicles navigating traffic jams 🚗💨, to optimizing supply chains for faster deliveries with less waste 📦🌍, and even helping doctors make better healthcare decisions 🏥💡, RL is making a tangible impact. But here's a critical question: how do we balance our desire for smarter machines with the urgent need to protect the environment? RL is proving to be a powerful tool that pushes AI forward sustainably, reducing resource use without compromising performance. 🌿⚖️

For those of you experimenting with RL, here’s a bonus tip: try integrating randomized neural network layers into your models. This might be the secret sauce to slash your training costs while boosting efficiency. Sometimes, a little randomness leads to smarter, leaner AI. 🎲💡

To close, I’ll leave you with the wise words of Richard Feynman: “What I cannot create, I do not understand.” Reinforcement learning is helping us build AI that truly understands and adapts, all while respecting the limits of our computational resources. 🔧🤓

Stay connected with me on YouTube, Twitter, LinkedIn, and Medium for deeper insights and updates. Don’t forget to subscribe to AI with Shaily for your weekly dose of AI news. I’d love to hear your thoughts on the push for AI efficiency—drop a comment and let’s keep the conversation alive! 💬✨

Thanks for tuning in! Remember, smarter AI doesn’t have to mean bigger energy bills. I’m Shailendra Kumar, and this was AI with Shaily. See you next time! 👋🤖🌟

Reinforcement Learning Revolution: Smarter AI with Less Energy

We discuss Compiler-R1, a novel AI system designed to optimize code efficiency by intelligently auto-tuning compilers. It highlights the challenges of traditional, manual code optimization and the escalating costs associated with inefficient cloud computing. Compiler-R1 leverages a two-stage approach—supervised fine-tuning followed by reinforcement learning—to discover optimal compiler pass sequences, leading to significant reductions in program instruction counts. The article also touches upon the broader trend of AI-driven code optimization, mentioning other research efforts like Pearl and DeepMind's work, and discusses the implications for CI/CD pipelines and the potential for substantial cost savings and improved user experience. Ultimately, the text suggests that automated optimization will become a competitive necessity, freeing engineers to focus on higher-level tasks.

CI/CD’s Final Frontier: Fully Autonomous Code Optimization

Welcome to "AI with Shaily" hosted by Shailendra Kumar 👨‍💻, where the latest breakthroughs and challenges in artificial intelligence are explored with depth and clarity. In this episode, Shailendra dives into a vital topic shaping the future of AI: the guardrails against misalignment in Artificial Superintelligence (Super AI) 🤖⚖️.

He paints a vivid picture of training a brilliant but unpredictable apprentice who learns rapidly and may soon outsmart even the brightest human minds. The core challenge is ensuring this apprentice shares human values and goals, especially as its capabilities grow beyond our full understanding. This dilemma is at the heart of AI research today.

As of 2025, there is an intense global effort to build multi-layered safety guardrails to keep Super AI aligned with human ethics and societal values 🌍🛡️. These safety measures begin early in the AI training process through techniques like supervised fine-tuning and reinforcement learning with human feedback (RLHF). This means AI models are not just released blindly; they undergo rigorous monitoring and repeated testing, including automated and manual “red teaming” exercises designed to uncover vulnerabilities or harmful behaviors before deployment.

However, Shailendra emphasizes that alignment is not a one-time achievement but a continuously moving target 🎯. As AI systems grow more complex and capable, maintaining alignment demands ongoing vigilance, layered safety checks, and evolving guardrails that adapt alongside the AI’s development. While RLHF accelerates progress toward advanced AI, it also introduces new risks that must be carefully managed.

Beyond the technical side, Shailendra highlights the importance of societal-level guardrails such as transparent governance, strong regulatory frameworks, and ethical oversight 🏛️⚖️. These pillars are crucial to preventing fairness violations and even existential risks. Leading AI experts advocate for a precautionary approach, including potential pauses on massive AI training efforts until their societal impacts and safety can be fully understood.

Drawing from his personal experience in AI, Shailendra compares these guardrails to mentoring a prodigy musician 🎼🎻. It’s not just about teaching skills but nurturing ethics, discipline, and social responsibility—a holistic approach critical to AI development.

He also offers a practical tip for AI practitioners: integrate continual human feedback loops and transparency protocols early in projects rather than waiting for problems to appear after deployment. This proactive approach helps avoid costly ethical and technical issues later on 💡🔄.

To close, Shailendra shares a profound quote from AI theorist Stuart Russell: “The real problem with superintelligence isn’t machines beating us at chess. It’s that we’re playing the wrong game.” This thought invites reflection on how we approach AI’s future.

Shailendra invites viewers to stay connected via YouTube, Twitter, LinkedIn, and Medium under “AI with Shaily” for more insightful AI discussions. He encourages subscribing and sharing thoughts on balancing AI innovation with safety. His closing message inspires everyone to keep questioning, learning, and steering AI toward a future that benefits all humanity 🌟🤝.

In summary, this content is a thoughtful and comprehensive exploration of the challenges and strategies involved in aligning powerful AI systems with human values, emphasizing both technical and societal solutions, personal insights, and practical advice—all delivered with a warm, engaging tone by Shailendra Kumar, aka Shaily.

Guardrails for Super AI: Balancing Innovation and Safety in Artificial Superintelligence

Welcome to *AI with Shaily,* hosted by Shailendra Kumar, where the exciting world of artificial intelligence is explored with enthusiasm and insight. Today’s episode dives into the thrilling competition between human instinctive movement and the remarkable progress of physical AI robots mastering adaptive locomotion. Whether you’re lacing up your running shoes or charging robot batteries, this is a race full of surprises! 🏃‍♂️🤖⚡

Shailendra begins with an intriguing story about the Chinese humanoid robot Unite G1, which stunned viewers by performing kung fu moves and flips with fluid, dance-like grace. This impressive agility is powered by Nvidia’s Isaac simulator and reinforcement learning, where the robot is trained in a virtual environment before transferring those skills into the real world. The robot’s software is updated monthly, akin to teaching it to ride a bike and then upgrading it to do wheelies! 🥋🤸‍♂️🖥️

The discussion then highlights a groundbreaking event in Beijing: the first-ever humanoid robot half-marathon, where 21 robots attempted to run 21 kilometers. Only 4 finished due to crashes and mechanical glitches, illustrating the current limitations of robots when faced with unpredictable real-world challenges. Unlike humans who instinctively adjust to obstacles like puddles or slippery slopes, robots still struggle with balance and adaptability. 🏅🤖💥

Among the standout performers is the Star One robot, which impressed with a top speed of 13 kilometers per hour—faster than many casual joggers—and the ability to dynamically adjust its gait over challenging terrains such as sand and rocks. This showcases the aspiration for robots to run trail marathons smoothly, maintaining stride despite diverse surfaces. Meanwhile, Boston Dynamics’ Atlas robot continues to push boundaries by learning from human motion capture, executing walking, running, crawling, and athletic moves. Reinforcement learning enables these robots to become real-time learners, constantly refining their movements. 🏃‍♂️🌄🤖✨

The key takeaway is that AI-powered robots are rapidly advancing in agility, making humanoid movement increasingly realistic. However, they remain like children learning to walk, run, and dance, while humans are seasoned athletes and parkour experts. The instinctive adaptability that humans possess is still a major hurdle for these mechanical marvels. 👶🤖🏃‍♂️💨

Shailendra offers a bonus tip for AI and robotics enthusiasts: watch simulation environments like Nvidia’s Isaac closely. These virtual training grounds are where robots learn movement safely and efficiently before entering the physical world—think of it as the ultimate training camp without the risk of injury or damage. 🎮🦾🏋️‍♂️

To spark further thought, Shailendra poses a compelling question: If robots one day master not just movement but the intuition behind it, how will this transform industries such as healthcare, disaster relief, or everyday life? Could robots even take over mundane tasks like running errands? 🤔🏥🚒🛒

The episode closes with a powerful quote from robotics pioneer Marc Raibert: “Balance is the essence of motion.” This perfectly captures the current state of AI robotics—poised on the brink of an extraordinary leap forward, balancing literal and figurative challenges. ⚖️🤖🚀

Shailendra Kumar signs off warmly, inviting listeners to connect on YouTube, Twitter, LinkedIn, and Medium for more AI insights. He encourages everyone to subscribe, join the conversation, and share their thoughts, promising to read all comments. Until next time, keep curious and keep moving forward! 🌟📱💬

AI with Shaily: The Race to Teach Robots Human-Like Movement and Balance

A structured, reflective overview of how episodic memory and recursive models allow both humans and AI systems to simulate, refine, and evolve.
This technical perspective connects biological memory loops (e.g. in the hippocampus and DMN) with recursive processes in artificial architectures — such as reinforcement learning, feedback modelling, and emergent simulation.
Delivered in Arty’s grounded tone, this track continues our exploration of intelligent memory as the foundation for intentional design.

Chapter 23 Section 2, Technical Perspective

A structured analysis of the instability inherent in emergent intention — from neurobiology to artificial cognition.
This Technical Perspective maps the complex pathways between stimulus and reflective choice, referencing how fragile intention arises from competing subsystems in the brain and AI architectures.
It explores prefrontal-limbic dynamics, decision delays, and the risk of collapse in goal-oriented systems lacking memory depth or error correction.
A key resource in understanding reflective design and intention modelling in conscious systems.

Chapter 23, Section 3. The Fragility of Intention. Technical Perspective

Welcome back to *AI with Shaily*! I’m Shailendra Kumar, your host, here to take you on an exciting journey into the world of artificial intelligence 🤖✨. Today, we’re exploring Google DeepMind’s groundbreaking innovation called AlphaChip—a technology that’s causing a buzz on social media and revolutionizing hardware engineering 🔥💻.

Imagine designing a processor chip as if you’re playing a strategic game 🎲. AlphaChip uses reinforcement learning to do just that. Starting from a blank slate, it places circuit components one by one, learning from successes and failures. If a design performs well, it earns rewards; if not, it faces penalties. Think of it like AI playing chess, but instead of chess pieces, it’s arranging transistors and wires. With every attempt, AlphaChip gets smarter and faster at crafting efficient chip layouts 🧠♟️.

A standout feature of AlphaChip is its use of an “edge-based” graph neural network. This means the AI understands how each chip component connects within the entire system, allowing it to adapt across different chip types without starting over every time 🔄🔗. For engineers, this is a revolutionary leap forward!

Here’s the game changer: traditional chip floorplan design can take months or even up to two years for complex GPUs ⏳. AlphaChip slashes that timeline to just a few hours ⚡. This not only saves millions in costs but also produces designs that outperform human experts by optimizing power efficiency, space, and performance simultaneously. It’s like upgrading from a bicycle to a Tesla 🚲➡️🚗⚡.

From my own experience in AI development, optimization once felt like a mix of guesswork and patience. Watching AlphaChip accelerate these tasks makes me marvel at how fast AI is evolving—and it sparks curiosity about what other “impossible” engineering challenges AI might soon conquer 🚀🤔.

AlphaChip isn’t just theoretical; it powers Google’s latest Tensor Processing Units—the AI engines behind Gemini language models and Imagen image generators you might have heard about 🧩🖼️. Plus, MediaTek is integrating it into their Dimensity 5G chips, meaning this tech is reaching millions of smartphones globally 📱🌍. Google has even shared a pre-trained model checkpoint with researchers worldwide, fostering open innovation and collaboration 🌐🤝.

Why should you care? Beyond the tech specs, AlphaChip signals a future where devices are smaller, faster, and more energy-efficient—impacting everything from your smartphone to AI in healthcare and autonomous vehicles 🚗🏥📱. Reinforcement learning is no longer just a lab experiment; it’s transforming semiconductor design after six decades of mostly manual work 🔧⚙️.

For AI enthusiasts, here’s a bonus tip: when tackling reinforcement learning, think of your problem as a “game” with clear rewards and penalties. This mindset can turn complex optimization into manageable challenges, just like AlphaChip’s success story 🎯🎮.

I’ll leave you with a wise quote from AI pioneer Marvin Minsky: “You don’t understand anything until you learn it more than one way.” AlphaChip embodies this by relearning chip design from a fresh perspective to achieve extraordinary results 🌟🔍.

For more AI insights and updates, follow me on YouTube, Twitter, LinkedIn, and Medium. If you enjoyed this episode or have thoughts to share, please subscribe and comment—let’s keep the AI conversation alive! 💬📲

Until next time, this is Shailendra Kumar signing off from *AI with Shaily*. What other industries do you think AI will reinvent next? Think about it, and I’ll catch you soon! 👋🤖

AlphaChip Unleashed: How AI is Revolutionizing Chip Design in Hours, Not Years

Unlock the secrets of data mastery with Shailendra, a 27-year veteran in AI, ML, and Data Science, driving AU$ 6 Billion in transformative value. As the author of 'Making Money out of Data,' their strategies redefine digital success. With expertise honed at IBM and Accenture, they drive innovation and data monetization. Explore how loyalty programs like Flybuys and Everyday Rewards revolutionize business—connect with Shaily today.

Software engineers, architects and team leads have found inspiration to drive change and innovation in their team by listening to the weekly InfoQ Podcast. They have received essential information that helped them validate their software development map. We have achieved that by interviewing some of the top CTOs, engineers and technology directors from companies like Uber, Netflix and more. Over 1,200,000 downloads in the last 3 years.

The latest machine learning, A.I., and data career topics from across both academia and industry are brought to you by host Dr. Jon Krohn on the Super Data Science Podcast.

As the quantity of data on our planet doubles every couple of years and with this trend set to continue for decades to come, there's an unprecedented opportunity for you to make a meaningful impact in your lifetime. In conversation with the biggest names in the data science industry, Jon cuts through hype to fuel that professional impact.

Whether you're curious about getting started in a data career or you're a deep technical expert, whether you'd like to understand what A.I. is or you'd like to integrate more data-driven processes into your business, we have inspiring guests and lighthearted conversation for you to enjoy.

We cover tools, techniques, and implementation tricks across data collection, databases, analytics, predictive modeling, visualization, software engineering, real-world applications, commercialization, and entrepreneurship − everything you need to crush it with data science.

I'm machine learning enthusiast with applied math background and a lot experience as full stack developer.
I'm from Ukraine but now I'm working remotely in Montenegro.

New tracks tagged #Reinforcement Learning