What function do the different attention heads serve in multi-headed attention models?  In this episode, Lena describes how to use attribution methods to assess the importance and contribution of different heads in several tasks, and describes a gating mechanism to prune the number of effective heads used when combined with an auxiliary loss. Then, we discuss Lena’s work on studying the evolution of representations of individual tokens in transformers model.

Lena’s homepage:
https://lena-voita.github.io/

Blog posts:
https://lena-voita.github.io/posts/acl19_heads.html
https://lena-voita.github.io/posts/emnlp19_evolution.html

Papers:
https://arxiv.org/abs/1905.09418
https://arxiv.org/abs/1909.01380

98 - Analyzing Information Flow In Transformers, With Elena Voita

**The podcast is currently on hiatus. For more active NLP content, check out the Holistic Intelligence Podcast linked below.**

Welcome to the NLP highlights podcast, where we invite researchers to talk about their work in various areas in natural language processing. All views expressed belong to the hosts/guests, and do not represent their employers.

Founder of EPIC Knowledge Society and SPACE Creative Culture Labs. I am a Futurist and Data Scientist with Professional Mastery and Pragmatic Insights on Digital Transformation and Innovation Management with extensive credentials on Digital Enterprise Architectures, Crypto Economics, Customer Experience Design, Social Entrepreneurship Incubation, Business Model Innovation, Innovation Ecosystems, Innovation Analytics, Innovation Consulting, Social Business Networks, Academic Research Networks, Industrial Research Alliances, New Service Incubation, New Product Incubation, New Platform Innovation, High performing Teams, Cross Competency Innovation etc.

98 - Analyzing Information Flow In Transformers, With Elena Voita

NLP Highlights