Encoders, cross attention and masking for LLMs: SuperDataScience Founder Kirill Eremenko returns to the SuperDataScience podcast, where he speaks with Jon Krohn about transformer architectures and why they are a new frontier for generative AI. If you’re interested in applying LLMs to your business portfolio, you’ll want to pay close attention to this episode!

This episode is brought to you by Ready Tensor, where innovation meets reproducibility (https://www.readytensor.ai/), by Oracle NetSuite business software (netsuite.com/superdata), and by Intel and HPE Ezmeral Software Solutions (http://hpe.com/ezmeral/chatbots). Interested in sponsoring a SuperDataScience Podcast episode? Email natalie@superdatascience.com for sponsorship information.

In this episode you will learn:
• How decoder-only transformers work [15:51]
• How cross-attention works in transformers [41:05]
• How encoders and decoders work together (an example) [52:46]
• How encoder-only architectures excel at understanding natural language [1:20:34]
• The importance of masking during self-attention [1:27:08]

Additional materials: www.superdatascience.com/759

The Super Data Science Podcast: Machine Learning | AI | Success

759: Full Encoder-Decoder Transformers Fully Explained, with Kirill Eremenko

Discover how Generative AI creates unique content and how RAG uses real-time data for precision. Explore essential tips, tools, and steps now.

how generative ai works

Fresh out of the studio, Doug Burger, Technical Fellow and Corporate Vice President at Microsoft Research, joins us to explore Microsoft's bold expansion into Southeast Asia with the recent launch of the Microsoft Research Asia lab in Singapore. From there, Doug shares his accidental journey from academia to leading global research operations, reflecting on how Microsoft Research's open collaboration model empowers over thousands of researchers worldwide to tackle humanity's biggest challenges. Addressing the intersection of three computing paradigms: logic, probability, and quantum, he emphasizes that geographic diversity in research labs enables Microsoft to build AI that works for everyone. Closing the conversation, Doug shares his vision of what great looks like for Microsoft Research with researchers driven by purpose and passion to create breakthroughs that advance both science and society.

"If you're going to be running a very elite research institution, you have to have the best people. To have the best people, you have to trust them and empower them. You can't hire a world expert in some area and then tell them what to do. They know more than you do. They're smarter than you are in their area. So you've got to trust your people. One of our really foundational commitments to our people is: we trust you. We're going to work to empower you. Go do the thing that you need to do. If somebody in the labs wants to spend 5, 10, 15 years working on something they think is really important, they're empowered to do that."

Episode Highlights:
[00:00] Quote of the Day by Doug Burger 
[01:08] Doug Burger's journey from academia to Microsoft Research
[03:07] Launch of Microsoft Research Asia in Singapore: Tapping local talent and culture for inclusive AI development
[04:13] Singapore lab focuses on foundational AI, embodied AI, and healthcare applications
[06:19] AI detecting seizures in children and assessing Parkinson's motor function
[08:24] Embedding Southeast Asian societal norms & values into Foundational AI research
[10:26] Microsoft Research's open collaboration model 
[12:42] Generative AI's rapid pace accelerating technological innovation and research tools
[16:24] Open versus closed source AI models debate and Microsoft's platform approach
[19:35] Multiple scaling dimensions in AI beyond traditional data scaling laws
[26:26] Balancing blue-sky foundational research with application-driven initiatives at scale
[35:26] Research leadership secret: Curiosity across fields enables unexpected connections
[40:04] Microsoft Research's vision: Becoming Bell Labs for AI era
[42:22] Steering AI models for mental health and critical thinking conversations

How Microsoft Research Balances Exploration and Impact Globally with Doug Burger

A weekly podcast dissecting the pulse of business, technology & media in Asia. Hear our host interviewing leading journalists, senior executives, respectable authors & thinkers, entrepreneurs & thought leaders to discuss the most important issues that move and shake the business landscape in the Asia Pacific. The show is originally created and hosted by Bernard Leong.

The latest machine learning, A.I., and data career topics from across both academia and industry are brought to you by host Dr. Jon Krohn on the Super Data Science Podcast.

As the quantity of data on our planet doubles every couple of years and with this trend set to continue for decades to come, there's an unprecedented opportunity for you to make a meaningful impact in your lifetime. In conversation with the biggest names in the data science industry, Jon cuts through hype to fuel that professional impact.

Whether you're curious about getting started in a data career or you're a deep technical expert, whether you'd like to understand what A.I. is or you'd like to integrate more data-driven processes into your business, we have inspiring guests and lighthearted conversation for you to enjoy.

We cover tools, techniques, and implementation tricks across data collection, databases, analytics, predictive modeling, visualization, software engineering, real-world applications, commercialization, and entrepreneurship − everything you need to crush it with data science.

New tracks tagged #transformer architecture