In this episode, we invite Hao Tan and Mohit Bansal to talk about multi-modal training of transformers, focusing in particular on their EMNLP 2019 paper that introduced LXMERT, a vision+language transformer.  We spend the first third of the episode talking about why you might want to have multi-modal representations.  We then move to the specifics of LXMERT, including the model structure, the losses that are used to encourage cross-modal representations, and the data that is used.  Along the way, we mention latent alignments between images and captions, the granularity of captions, and machine translation even comes up a few times.  We conclude with some speculation on the future of multi-modal representations.

Hao's website: http://www.cs.unc.edu/~airsplay/
Mohit's website: http://www.cs.unc.edu/~mbansal/
LXMERT paper: https://www.aclweb.org/anthology/D19-1514/

107 - Multi-Modal Transformers, with Hao Tan and Mohit Bansal

**The podcast is currently on hiatus. For more active NLP content, check out the Holistic Intelligence Podcast linked below.**

Welcome to the NLP highlights podcast, where we invite researchers to talk about their work in various areas in natural language processing. All views expressed belong to the hosts/guests, and do not represent their employers.

Founder of EPIC Knowledge Society and SPACE Creative Culture Labs. I am a Futurist and Data Scientist with Professional Mastery and Pragmatic Insights on Digital Transformation and Innovation Management with extensive credentials on Digital Enterprise Architectures, Crypto Economics, Customer Experience Design, Social Entrepreneurship Incubation, Business Model Innovation, Innovation Ecosystems, Innovation Analytics, Innovation Consulting, Social Business Networks, Academic Research Networks, Industrial Research Alliances, New Service Incubation, New Product Incubation, New Platform Innovation, High performing Teams, Cross Competency Innovation etc.

107 - Multi-Modal Transformers, with Hao Tan and Mohit Bansal

NLP Highlights