Awesome Podcast

later

In this episode, Robin Jia talks about how to build robust NLP systems. We discuss the different senses in which a system can be robust, reasons to care about system robustness, and the challenges involved in evaluating robustness of NLP models. We talk about how to build certifiably robust models through interval bound propagation and discrete encoding functions, as well as how to modify data collection procedures through active learning for more robust model development.

Robin Jia is currently a visiting researcher at Facebook AI Research, and will be an assistant professor in the Department of Computer Science at the University of Southern California starting Fall 2021.

123 - Robust NLP, with Robin Jia

We invited Jayant Krishnamurthy and Hao Fang, researchers at Microsoft Semantic Machines to discuss their platform for building task-oriented dialog systems, and their recent TACL paper on the topic. The paper introduces a new formalism for task-oriented dialog to effectively handle references and revisions in complex dialog, and a large realistic dataset that uses this formalism.

Leaderboard associated with the dataset: https://microsoft.github.io/task_oriented_dialogue_as_dataflow_synthesis/
Jayant's Twitter handle: https://twitter.com/jayantkrish
Hao's Twitter handle: https://twitter.com/hfang90

124 - Semantic Machines and Task-Oriented Dialog, with Jayant Krishnamurthy and Hao Fang

How can we build Visual Question Answering systems for real users? For this episode, we chatted with Danna Gurari, about her work in building datasets and models towards VQA for people who are blind. We talked about the differences between the existing datasets, and Vizwiz, a dataset built by Gurari et al., and the resulting algorithmic changes. We also discussed the unsolved challenges in this field, and the new tasks they result in.

Danna Gurari is an Assistant Professor as well as Founding Director of the Image and Video Computing group in the School of Information at University of Texas at Austin (UT-Austin).

Vizwiz project page: https://vizwiz.org/

The hosts for this episode are Ana Marasović and Pradeep Dasigi.

125 - VQA for Real Users, with Danna Gurari

We invited Lisa Li to talk about her recent work, Prefix-Tuning: Optimizing Continuous Prompts for Generation. Prefix tuning is a lightweight alternative to finetuning, and the idea is to tune only a fixed-length task-specific continuous vector, and to keep the pretrained transformer parameters frozen. We discussed how prefix tuning compares with finetuning and other efficient alternatives on two tasks in various experimental settings, and in what scenarios prefix tuning is preferable.

Lisa is a Phd student at Stanford University. Lisa's webpage: https://xiangli1999.github.io/

The hosts for this episode are Pradeep Dasigi and Ana Marasović.

126 - Optimizing Continuous Prompts for Generation, with Lisa Li

In this episode, we talk to Shunyu Yao about recent insights into how transformers can represent hierarchical structure in language. Bounded-depth hierarchical structure is thought to be a key feature of natural languages, motivating Shunyu and his coauthors to show that transformers can efficiently represent bounded-depth Dyck languages, which can be thought of as a formal model of the structure of natural languages. We went on to discuss some of the intuitive ideas that emerge from the proofs, connections to RNNs, and insights about positional encodings that may have practical implications. More broadly, we also touched on the role of formal languages and other theoretical tools in modern NLP.

Papers discussed in this episode:

- Self-Attention Networks Can Process Bounded Hierarchical Languages (https://arxiv.org/abs/2105.11115)
- Theoretical Limitations of Self-Attention in Neural Sequence Models (https://arxiv.org/abs/1906.06755)
- RNNs can generate bounded hierarchical languages with optimal memory (https://arxiv.org/abs/2010.07515)
- On the Practical Computational Power of Finite Precision RNNs for Language Recognition (https://arxiv.org/abs/1805.04908)

Shunyu Yao's webpage: https://ysymyth.github.io/

The hosts for this episode are William Merrill and Matt Gardner.

129 - Transformers and Hierarchical Structure, with Shunyu Yao

In this episode we briefly say what we're up to with the podcast.  No technical content, just a description of what each episode will look like, and why we're doing this.

00 - Intro to the podcast

https://www.semanticscholar.org/paper/A-Comparative-Study-of-Word-Embeddings-for-Reading-Dhingra-Liu/3ec37205c9201fc891ab51da200e361fdc34bfb3

01 - A Comparative Study of Word Embeddings for Reading Comprehension

https://www.semanticscholar.org/paper/Bidirectional-Attention-Flow-for-Machine-Seo-Kembhavi/007ab5528b3bd310a80d553cccad4b78dc496b02

02 - Bidirectional Attention Flow for Machine Comprehension

https://www.semanticscholar.org/paper/FastQA-A-Simple-and-Efficient-Neural-Architecture-Weissenborn-Wiese/7c1576b96a1e246d77b30f7b80cec63be96fa698

03 - FastQA: A Simple and Efficient Neural Architecture for Question Answering

An interview with Chris Dyer.

https://www.semanticscholar.org/paper/Recurrent-Neural-Network-Grammars-Dyer-Kuncoro/1594d954abc650bce2db445c52a76e49655efb0c

04 - Recurrent Neural Network Grammars, with Chris Dyer

ACL 2017 paper by Abigail See, Peter Liu, and Chris Manning.

Matt presents the paper, describing the task (summarization on CNN/Daily Mail), the model (the standard copy + generate model that people are using these days, plus a nice coverage loss term), and the results (can't beat the extractive baseline, but coming close).  It's a nice paper - very well written, interesting discussion section.

https://www.semanticscholar.org/paper/Get-To-The-Point-Summarization-with-Pointer-Genera-See-Liu/13db673d09f546698e0bfb6687beeb5345f81ad9

Abigail also has a very nice blog post where she describes her work in a less formal tone than the paper: http://www.abigailsee.com/2017/04/16/taming-rnns-for-better-summarization.html

23 - Get To The Point: Summarization with Pointer-Generator Networks

NIPS 2017 paper.

We dig into the details of the Transformer, from the "attention is all you need" paper.  Ashish and Jakob give us some motivation for replacing RNNs and CNNs with a more parallelizable self-attention mechanism, they describe how this mechanism works, and then we spend the bulk of the episode trying to get their intuitions for _why_ it works.  We discuss the positional encoding mechanism, multi-headed attention, trying to use these ideas to replace encoders in other models, and what the self-attention actually learns.  Turns out that the lower layers learn something like n-grams (similar to CNNs), and the higher layers learn more semantic-y things, like coreference.

https://www.semanticscholar.org/paper/Attention-Is-All-You-Need-Vaswani-Shazeer/0737da0767d77606169cbf4187b83e1ab62f6077

Minor correction: Talking about complexity equations without the paper in front of you can be tricky, and Ashish and Jakob may have gotten some of the details slightly wrong when we were discussing computational complexity.  The high-level point is that self-attention is cheaper than RNNs when the hidden dimension is higher than the sequence length.  See the paper for more details.

36 - Attention Is All You Need, with Ashish Vaswani and Jakob Uszkoreit

ACL 2017 paper, by Paramita Mirza, Simon Razniewski, Fariz Darari, and Gerhard Weikum.

There's not a whole lot of work on numbers in NLP, and getting good information out of numbers expressed in text can be challenging.  In this episode, Paramita comes on to tell us about her efforts to use distant supervision to learn models that extract relation cardinalities from text.  That is, given an entity and a relation in a knowledge base, like "Barack Obama" and "has child", the goal is to extract _how many_ related entities there are (in this case, two).  There are a lot of challenges in getting this to work well, and Paramita describes some of those, and how she solved them.

https://www.semanticscholar.org/paper/Cardinal-Virtues-Extracting-Relation-Cardinalities-Mirza-Razniewski/01afba9f40e0df06446b9cd3d5ea8725c4ba1342

50 - Cardinal Virtues: Extracting Relation Cardinalities from Text, with Paramita Mirza

NAACL 2018 paper, by Matt Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Chris Clark, Kenton Lee, and Luke Zettlemoyer.

In this episode, AI2's own Matt Peters comes on the show to talk about his recent work on ELMo embeddings, what some have called "the next word2vec".  Matt has shown very convincingly that using a pre-trained bidirectional language model to get contextualized word representations performs substantially better than using static word vectors.  He comes on the show to give us some more intuition about how and why this works, and to talk about some of the other things he tried and what's coming next.

https://www.semanticscholar.org/paper/Deep-contextualized-word-representations-Peters-Neumann/4b17597b856c087f109381ce77d60d9017cb6f9a

NLP Highlights

56 - Deep contextualized word representations, with Matthew Peters

Upcoming JAIR paper by Sebastian Ruder, Ivan Vulić, and Anders Søgaard.

Sebastian comes on to tell us about his survey.  He creates a typology of cross-lingual word embedding methods, and we discuss why you might use cross-lingual embeddings (low-resource languages in particular), what information they capture (semantics? syntax? both?), how the methods work (lots of different ways), and how to evaluate the embeddings (best when you have an extrinsic task to evaluate on).

https://www.semanticscholar.org/paper/A-survey-of-cross-lingual-embedding-models-Ruder/3dbd28c63a7807280c9531735c715d4598024166

Episode 57_Sebastian Ruder

57 - A Survey Of Cross-lingual Word Embedding Models, with Sebastian Ruder

In this episode we talked with Victor Sanh and Thomas Wolf from HuggingFace about model distillation, and DistilBERT as one example of distillation. The idea behind model distillation is compressing a large model by building a smaller model, with much fewer parameters, that approximates the output distribution of the original model, typically for increased efficiency. We discussed how model distillation was typically done previously, and then focused on the specifics of DistilBERT, including training objective, empirical results, ablations etc. We finally discussed what kinds of information you might lose when doing model distillation.

104 - Model Distillation, with Victor Sanh and Thomas Wolf

**The podcast is currently on hiatus. For more active NLP content, check out the Holistic Intelligence Podcast linked below.**

Welcome to the NLP highlights podcast, where we invite researchers to talk about their work in various areas in natural language processing. All views expressed belong to the hosts/guests, and do not represent their employers.