In this episode we talked with Victor Sanh and Thomas Wolf from HuggingFace about model distillation, and DistilBERT as one example of distillation. The idea behind model distillation is compressing a large model by building a smaller model, with much fewer parameters, that approximates the output distribution of the original model, typically for increased efficiency. We discussed how model distillation was typically done previously, and then focused on the specifics of DistilBERT, including training objective, empirical results, ablations etc. We finally discussed what kinds of information you might lose when doing model distillation.

104 - Model Distillation, with Victor Sanh and Thomas Wolf

**The podcast is currently on hiatus. For more active NLP content, check out the Holistic Intelligence Podcast linked below.**

Welcome to the NLP highlights podcast, where we invite researchers to talk about their work in various areas in natural language processing. All views expressed belong to the hosts/guests, and do not represent their employers.

Founder of EPIC Knowledge Society and SPACE Creative Culture Labs. I am a Futurist and Data Scientist with Professional Mastery and Pragmatic Insights on Digital Transformation and Innovation Management with extensive credentials on Digital Enterprise Architectures, Crypto Economics, Customer Experience Design, Social Entrepreneurship Incubation, Business Model Innovation, Innovation Ecosystems, Innovation Analytics, Innovation Consulting, Social Business Networks, Academic Research Networks, Industrial Research Alliances, New Service Incubation, New Product Incubation, New Platform Innovation, High performing Teams, Cross Competency Innovation etc.

104 - Model Distillation, with Victor Sanh and Thomas Wolf

NLP Highlights