Awesome Podcast

In this episode we briefly say what we're up to with the podcast.  No technical content, just a description of what each episode will look like, and why we're doing this.

00 - Intro to the podcast

https://www.semanticscholar.org/paper/A-Comparative-Study-of-Word-Embeddings-for-Reading-Dhingra-Liu/3ec37205c9201fc891ab51da200e361fdc34bfb3

01 - A Comparative Study of Word Embeddings for Reading Comprehension

https://www.semanticscholar.org/paper/Bidirectional-Attention-Flow-for-Machine-Seo-Kembhavi/007ab5528b3bd310a80d553cccad4b78dc496b02

02 - Bidirectional Attention Flow for Machine Comprehension

https://www.semanticscholar.org/paper/FastQA-A-Simple-and-Efficient-Neural-Architecture-Weissenborn-Wiese/7c1576b96a1e246d77b30f7b80cec63be96fa698

03 - FastQA: A Simple and Efficient Neural Architecture for Question Answering

An interview with Chris Dyer.

https://www.semanticscholar.org/paper/Recurrent-Neural-Network-Grammars-Dyer-Kuncoro/1594d954abc650bce2db445c52a76e49655efb0c

04 - Recurrent Neural Network Grammars, with Chris Dyer

What function do the different attention heads serve in multi-headed attention models?  In this episode, Lena describes how to use attribution methods to assess the importance and contribution of different heads in several tasks, and describes a gating mechanism to prune the number of effective heads used when combined with an auxiliary loss. Then, we discuss Lena’s work on studying the evolution of representations of individual tokens in transformers model.

Lena’s homepage:
https://lena-voita.github.io/

Blog posts:
https://lena-voita.github.io/posts/acl19_heads.html
https://lena-voita.github.io/posts/emnlp19_evolution.html

Papers:
https://arxiv.org/abs/1905.09418
https://arxiv.org/abs/1909.01380

98 - Analyzing Information Flow In Transformers, With Elena Voita

We talked to Brendan O’Connor for this episode about processing language in social media. Brendan started off by telling us about his projects that studied the linguistic and geographical patterns of African American English (AAE), and how obtaining data from Twitter made these projects possible. We then talked about how many tools built for standard English perform very poorly on AAE, and why collecting dialect-specific data is important. For the rest of the conversation, we discussed the issues involved in scraping data from social media, including ethical considerations and the biases that the data comes with.

Brendan O’Connor is an Assistant Professor at the University of Massachusetts, Amherst.

Warning: This episode contains explicit language (one swear word).

103 - Processing Language in Social Media, with Brendan O'Connor

In this episode, we talked to Emily Bender about the ethical considerations in developing NLP models and putting them in production. Emily cited specific examples of ethical issues, and talked about the kinds of potential concerns to keep in mind, both when releasing NLP models that will be used by real people, and also while conducting NLP research. We concluded by discussing a set of open-ended questions about designing tasks, collecting data, and publishing results, that Emily has put together towards addressing these concerns.

Emily M. Bender is a Professor in the Department of Linguistics and an Adjunct Professor in the Department of Computer Science and Engineering at the University of Washington. She’s active on Twitter at @emilymbender.

106 - Ethical Considerations In NLP Research, with Emily Bender

In this episode we invite Verena Rieser and Ondřej Dušek on to talk to us about the complexities of generating natural language when you have some kind of structured meaning representation as input.  We talk about when you might want to do this, which is often is some kind of a dialog system, but also generating game summaries, and even some language modeling work.  We then talk about why this is hard, which in large part is due to the difficulty of collecting data, and how to evaluate the output of these systems.

We then move on to discussing the details of a major challenge that Verena and Ondřej put on, called the end-to-end natural language generation challenge (E2E NLG). This was a dataset of task-based dialog generation focused on the restaurant domain, with some very innovative data collection techniques.  They held a shared task with 16 participating teams in 2017, and the data has been further used since.  We talk about the methods that people used for the task, and what we can learn today from what methods have been used on this data.

Verena's website: https://sites.google.com/site/verenateresarieser/
Ondřej's website: https://tuetschek.github.io/
The E2E NLG Challenge that we talked about quite a bit: http://www.macs.hw.ac.uk/InteractionLab/E2E/

108 - Data-To-Text Generation, with Verena Rieser and Ondřej Dušek

How do we know, in a concrete quantitative sense, what a deep learning model knows about language? In this episode, Ellie Pavlick talks about two broad directions to address this question: structural and behavioral analysis of models. In structural analysis, we often train a linear classifier for some linguistic phenomenon we'd like to probe (e.g., syntactic dependencies) while using the (frozen) weights of a model pre-trained on some tasks (e.g., masked language models). What can we conclude from the results of probing experiments? What does probing tell us about the linguistic abstractions encoded in each layer of an end-to-end pre-trained model? How well does it match classical NLP pipelines? How important is it to freeze the pre-trained weights in probing experiments? In contrast, behavioral analysis evaluates a model's ability to distinguish between inputs which respect vs. violate a linguistic phenomenon using acceptability or entailment tasks, e.g., can the model predict which is more likely: "dog bites man" vs. "man bites dog"? We discuss the significance of which format to use for behavioral tasks, and how easy it is for humans to perform such tasks.

Ellie Pavlick's homepage: 
https://cs.brown.edu/people/epavlick/

BERT rediscovers the classical nlp pipeline , by Ian Tenney, Dipanjan Das, Ellie Pavlick
https://arxiv.org/pdf/1905.05950.pdf?fbclid=IwAR3gzFibSBoDGdjqVu9Gq0mh1lDdRZa7dm42JuXXUfjG6rKZ44iHIOdV6jg

Inherent Disagreements in Human Textual Inferences by Ellie Pavlick and Tom Kwiatkowski 
https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00293

109 - What Does Your Model Know About Language, with Ellie Pavlick

**The podcast is currently on hiatus. For more active NLP content, check out the Holistic Intelligence Podcast linked below.**

Welcome to the NLP highlights podcast, where we invite researchers to talk about their work in various areas in natural language processing. All views expressed belong to the hosts/guests, and do not represent their employers.

106 - Ethical Considerations In NLP Research, with Emily Bender

NLP Highlights