In this episode, we invite John Hewitt to discuss his take on how to probe word embeddings for syntactic information. The basic idea is to project word embeddings to a vector space where the L2 distance between a pair of words in a sentence approximates the number of hops between them in the dependency tree. The proposed method shows that ELMo and BERT representations, trained with no syntactic supervision, embed many of the unlabeled, undirected dependency attachments between words in the same sentence.

Paper: https://nlp.stanford.edu/pubs/hewitt2019structural.pdf
GitHub repository: https://github.com/john-hewitt/structural-probes
Blog post: https://nlp.stanford.edu/~johnhew/structural-probe.html
Twitter thread: https://twitter.com/johnhewtt/status/1114252302141886464
John's homepage: https://nlp.stanford.edu/~johnhew/

88 - A Structural Probe for Finding Syntax in Word Representations, with John Hewitt

How is it like to do research in academia vs. industry? In this episode, we invite Jason Baldridge (UT Austin => Google) and Philip Resnik (Sun Microsystems => UMD) to discuss some of the aspects one may want to consider when planning their research careers, including flexibility, security and intellectual freedom. Perhaps most importantly, we discuss how the career choices we make influence and are influenced by the relationships we forge. Check out the Careers in NLP Panel at NAACL'19 on Monday, June 3, 2019 for further discussion.

Careers in NLP panel @ NAACL'19: https://naacl2019.org/blog/careers-panel-survey/
Jason Baldridge's homepage: http://www.jasonbaldridge.com/
Philip Resnik's homepage: http://users.umiacs.umd.edu/~resnik/

90 - Research in Academia versus Industry, with Philip Resnik and Jason Baldridge

What function do the different attention heads serve in multi-headed attention models?  In this episode, Lena describes how to use attribution methods to assess the importance and contribution of different heads in several tasks, and describes a gating mechanism to prune the number of effective heads used when combined with an auxiliary loss. Then, we discuss Lena’s work on studying the evolution of representations of individual tokens in transformers model.

Lena’s homepage:
https://lena-voita.github.io/

Blog posts:
https://lena-voita.github.io/posts/acl19_heads.html
https://lena-voita.github.io/posts/emnlp19_evolution.html

Papers:
https://arxiv.org/abs/1905.09418
https://arxiv.org/abs/1909.01380

98 - Analyzing Information Flow In Transformers, With Elena Voita

For our 100th episode, we invite AI2 CEO Oren Etzioni to talk to us about NLP startups.  Oren has founded several successful startups, is himself an investor in startups, and helps with AI2's startup incubator.

Some of our discussion topics include: What's the similarity between being a researcher and an entrepreneur?  How do you transition from being a researcher to doing a startup?  How do you evaluate early-stage startups?  What advice would you give to a researcher who's thinking about a startup?  What are some typical mistakes that you've seen startups make?  Along the way, Oren predicts a that we'll see a whole generation of startup companies based on the technology underlying ELMo, BERT, etc.

100 - NLP Startups, with Oren Etzioni

We talked to Brendan O’Connor for this episode about processing language in social media. Brendan started off by telling us about his projects that studied the linguistic and geographical patterns of African American English (AAE), and how obtaining data from Twitter made these projects possible. We then talked about how many tools built for standard English perform very poorly on AAE, and why collecting dialect-specific data is important. For the rest of the conversation, we discussed the issues involved in scraping data from social media, including ethical considerations and the biases that the data comes with.

Brendan O’Connor is an Assistant Professor at the University of Massachusetts, Amherst.

Warning: This episode contains explicit language (one swear word).

103 - Processing Language in Social Media, with Brendan O'Connor

In this episode, we talked to Emily Bender about the ethical considerations in developing NLP models and putting them in production. Emily cited specific examples of ethical issues, and talked about the kinds of potential concerns to keep in mind, both when releasing NLP models that will be used by real people, and also while conducting NLP research. We concluded by discussing a set of open-ended questions about designing tasks, collecting data, and publishing results, that Emily has put together towards addressing these concerns.

Emily M. Bender is a Professor in the Department of Linguistics and an Adjunct Professor in the Department of Computer Science and Engineering at the University of Washington. She’s active on Twitter at @emilymbender.

106 - Ethical Considerations In NLP Research, with Emily Bender

In this episode we invite Verena Rieser and Ondřej Dušek on to talk to us about the complexities of generating natural language when you have some kind of structured meaning representation as input.  We talk about when you might want to do this, which is often is some kind of a dialog system, but also generating game summaries, and even some language modeling work.  We then talk about why this is hard, which in large part is due to the difficulty of collecting data, and how to evaluate the output of these systems.

We then move on to discussing the details of a major challenge that Verena and Ondřej put on, called the end-to-end natural language generation challenge (E2E NLG). This was a dataset of task-based dialog generation focused on the restaurant domain, with some very innovative data collection techniques.  They held a shared task with 16 participating teams in 2017, and the data has been further used since.  We talk about the methods that people used for the task, and what we can learn today from what methods have been used on this data.

Verena's website: https://sites.google.com/site/verenateresarieser/
Ondřej's website: https://tuetschek.github.io/
The E2E NLG Challenge that we talked about quite a bit: http://www.macs.hw.ac.uk/InteractionLab/E2E/

108 - Data-To-Text Generation, with Verena Rieser and Ondřej Dušek

How do we know, in a concrete quantitative sense, what a deep learning model knows about language? In this episode, Ellie Pavlick talks about two broad directions to address this question: structural and behavioral analysis of models. In structural analysis, we often train a linear classifier for some linguistic phenomenon we'd like to probe (e.g., syntactic dependencies) while using the (frozen) weights of a model pre-trained on some tasks (e.g., masked language models). What can we conclude from the results of probing experiments? What does probing tell us about the linguistic abstractions encoded in each layer of an end-to-end pre-trained model? How well does it match classical NLP pipelines? How important is it to freeze the pre-trained weights in probing experiments? In contrast, behavioral analysis evaluates a model's ability to distinguish between inputs which respect vs. violate a linguistic phenomenon using acceptability or entailment tasks, e.g., can the model predict which is more likely: "dog bites man" vs. "man bites dog"? We discuss the significance of which format to use for behavioral tasks, and how easy it is for humans to perform such tasks.

Ellie Pavlick's homepage: 
https://cs.brown.edu/people/epavlick/

BERT rediscovers the classical nlp pipeline , by Ian Tenney, Dipanjan Das, Ellie Pavlick
https://arxiv.org/pdf/1905.05950.pdf?fbclid=IwAR3gzFibSBoDGdjqVu9Gq0mh1lDdRZa7dm42JuXXUfjG6rKZ44iHIOdV6jg

Inherent Disagreements in Human Textual Inferences by Ellie Pavlick and Tom Kwiatkowski 
https://www.mitpressjournals.org/doi/full/10.1162/tacl_a_00293

nlp

Peter Chochula