Awesome Podcast

Wouldn’t it solve so many problems in the world if everyone just spoke the same language? Not so fast! 

Lingthusiasm is a brand-new podcast that’s enthusiastic about linguistics, hosted by Lauren Gawne of Superlinguo​ and Gretchen McCulloch of All Things Linguistic.

In this first episode of Lingthusiasm, ​Gretchen and Lauren discuss the “one language equals peace” fallacy, and whether speaking the same words means that people will necessarily agree with each other (spoiler: no). But the history of how people have tried is still really interesting, from constructed and symbolic communication like Blissymbols and emoji to the way astronauts communicate in the high stakes environment of the International Space Station.

For links to things mentioned during this episode visit http://lingthusiasm.com/post/154520059101/lingthusiasm-episode-1-speaking-a-single-language

Listen to bonus episodes, suggest future topics, and help keep the show ad-free by supporting us on Patreon: http://patreon.com/lingthusiasm

01: Speaking a single language won’t bring about world peace

There are some pretty funny quotes of historical people complaining about kids back then doing linguistic things that now seem totally unremarkable. So let’s cut to the chase and celebrate linguistic innovation while it’s happening. 

In episode 7 of Lingthusiasm, your hosts Lauren Gawne and Gretchen McCulloch explore how far back we can trace complaints about the language of Kids These Days, why linguistic discrimination is harmful, and why “be like”, hyperbolic “literally”, and other modern innovations are actually signs of something awesome. 

We also announce a Patreon to keep the podcast sustainable! You can support us there to listen to a bonus episode about swearing and future monthly bonus content, and help decide the topics of future episodes. Even if you’re not sure about pledging, do check out the Patreon goals to see some of our future plans and our video to see a totally realistic and not at all staged cameo from our producer. Check out http://patreon.com/lingthusiasm 

For more information visit the show page: http://lingthusiasm.com/post/159796192161/lingthusiasm-episode-7-kids-these-days-arent

07: Kids these days aren’t ruining language

In this episode we briefly say what we're up to with the podcast.  No technical content, just a description of what each episode will look like, and why we're doing this.

00 - Intro to the podcast

https://www.semanticscholar.org/paper/A-Comparative-Study-of-Word-Embeddings-for-Reading-Dhingra-Liu/3ec37205c9201fc891ab51da200e361fdc34bfb3

01 - A Comparative Study of Word Embeddings for Reading Comprehension

https://www.semanticscholar.org/paper/Bidirectional-Attention-Flow-for-Machine-Seo-Kembhavi/007ab5528b3bd310a80d553cccad4b78dc496b02

02 - Bidirectional Attention Flow for Machine Comprehension

https://www.semanticscholar.org/paper/FastQA-A-Simple-and-Efficient-Neural-Architecture-Weissenborn-Wiese/7c1576b96a1e246d77b30f7b80cec63be96fa698

03 - FastQA: A Simple and Efficient Neural Architecture for Question Answering

An interview with Chris Dyer.

https://www.semanticscholar.org/paper/Recurrent-Neural-Network-Grammars-Dyer-Kuncoro/1594d954abc650bce2db445c52a76e49655efb0c

04 - Recurrent Neural Network Grammars, with Chris Dyer

How do we get from knowing words to making brand-new sentences out of them? In episode 9 of Lingthusiasm, your hosts Gretchen McCulloch and Lauren Gawne talk about how words form groups with other words: constituency. 

Once you start looking for it, constituency is everywhere: in ambiguous sentences like “time flies like an arrow; fruit flies like a banana”, in remixed films like “Of Oz The Wizard”, and even internet dog memes.

This month’s Patreon bonus was the backstory about the linguistics of the doggo meme and its connection to Australian slang, which grew out of this NPR article about doggo. You can get access to it and previous bonuses about swearing, teaching yourself linguistics, and explaining linguistics to employers by supporting Lingthusiasm on Patreon.

For more information, and links to things mentioned in this episode, visit the show page: http://lingthusiasm.com/post/161859273886/lingthusiasm-episode-9-the-bridge-between-words

09: The bridge between words and sentences - Constituency

ACL 2017 paper by Abigail See, Peter Liu, and Chris Manning.

Matt presents the paper, describing the task (summarization on CNN/Daily Mail), the model (the standard copy + generate model that people are using these days, plus a nice coverage loss term), and the results (can't beat the extractive baseline, but coming close).  It's a nice paper - very well written, interesting discussion section.

https://www.semanticscholar.org/paper/Get-To-The-Point-Summarization-with-Pointer-Genera-See-Liu/13db673d09f546698e0bfb6687beeb5345f81ad9

Abigail also has a very nice blog post where she describes her work in a less formal tone than the paper: http://www.abigailsee.com/2017/04/16/taming-rnns-for-better-summarization.html

23 - Get To The Point: Summarization with Pointer-Generator Networks

Some linguists work with multiple languages, while others focus on just one. But for many people, learning a language after early childhood is the thing that first gets them curious about how language works in general and all the things in their native language(s) that they take for granted. 

In episode 10 of Lingthusiasm, your hosts Gretchen McCulloch and Lauren Gawne talk about how learning languages can feed into linguistics and vice versa. We also explore the power dynamics that affect learning languages, and the importance of learning about the rules of interaction as well as the rules of grammar.

This month’s Patreon bonus was about hypercorrection, where you try so hard to follow a linguistic rule that you end up overshooting. You can get access to it and previous bonuses about the doggo meme, swearing, teaching yourself linguistics, and explaining linguistics to employers by supporting Lingthusiasm on Patreon. http://patreon.com/lingthusiasm

Gretchen’s new recorder in this episode is thanks to the support of our patrons! 

For more information, and links to things mentioned in this episode, visit the show page: http://lingthusiasm.com/post/163226798571/lingthusiasm-episode-10-learning-languages

10: Learning languages linguistically

NIPS 2017 paper.

We dig into the details of the Transformer, from the "attention is all you need" paper.  Ashish and Jakob give us some motivation for replacing RNNs and CNNs with a more parallelizable self-attention mechanism, they describe how this mechanism works, and then we spend the bulk of the episode trying to get their intuitions for _why_ it works.  We discuss the positional encoding mechanism, multi-headed attention, trying to use these ideas to replace encoders in other models, and what the self-attention actually learns.  Turns out that the lower layers learn something like n-grams (similar to CNNs), and the higher layers learn more semantic-y things, like coreference.

https://www.semanticscholar.org/paper/Attention-Is-All-You-Need-Vaswani-Shazeer/0737da0767d77606169cbf4187b83e1ab62f6077

Minor correction: Talking about complexity equations without the paper in front of you can be tricky, and Ashish and Jakob may have gotten some of the details slightly wrong when we were discussing computational complexity.  The high-level point is that self-attention is cheaper than RNNs when the hidden dimension is higher than the sequence length.  See the paper for more details.

36 - Attention Is All You Need, with Ashish Vaswani and Jakob Uszkoreit

ACL 2017 paper, by Paramita Mirza, Simon Razniewski, Fariz Darari, and Gerhard Weikum.

There's not a whole lot of work on numbers in NLP, and getting good information out of numbers expressed in text can be challenging.  In this episode, Paramita comes on to tell us about her efforts to use distant supervision to learn models that extract relation cardinalities from text.  That is, given an entity and a relation in a knowledge base, like "Barack Obama" and "has child", the goal is to extract _how many_ related entities there are (in this case, two).  There are a lot of challenges in getting this to work well, and Paramita describes some of those, and how she solved them.

https://www.semanticscholar.org/paper/Cardinal-Virtues-Extracting-Relation-Cardinalities-Mirza-Razniewski/01afba9f40e0df06446b9cd3d5ea8725c4ba1342

50 - Cardinal Virtues: Extracting Relation Cardinalities from Text, with Paramita Mirza

NAACL 2018 paper, by Matt Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Chris Clark, Kenton Lee, and Luke Zettlemoyer.

In this episode, AI2's own Matt Peters comes on the show to talk about his recent work on ELMo embeddings, what some have called "the next word2vec".  Matt has shown very convincingly that using a pre-trained bidirectional language model to get contextualized word representations performs substantially better than using static word vectors.  He comes on the show to give us some more intuition about how and why this works, and to talk about some of the other things he tried and what's coming next.

https://www.semanticscholar.org/paper/Deep-contextualized-word-representations-Peters-Neumann/4b17597b856c087f109381ce77d60d9017cb6f9a

NLP Highlights

56 - Deep contextualized word representations, with Matthew Peters

Upcoming JAIR paper by Sebastian Ruder, Ivan Vulić, and Anders Søgaard.

Sebastian comes on to tell us about his survey.  He creates a typology of cross-lingual word embedding methods, and we discuss why you might use cross-lingual embeddings (low-resource languages in particular), what information they capture (semantics? syntax? both?), how the methods work (lots of different ways), and how to evaluate the embeddings (best when you have an extrinsic task to evaluate on).

https://www.semanticscholar.org/paper/A-survey-of-cross-lingual-embedding-models-Ruder/3dbd28c63a7807280c9531735c715d4598024166

Episode 57_Sebastian Ruder

57 - A Survey Of Cross-lingual Word Embedding Models, with Sebastian Ruder

NAACL 2018 paper, by Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme

Rachel comes on to the podcast, telling us about what factuality is (did an event happen?), what datasets exist for doing this task (a few; they made a new, bigger one), and how to build models to predict factuality (turns out a vanilla biLSTM does quite well).  Along the way, we have interesting discussions about how you decide what an "event" is, how you label factuality (whether something happened) on inherently uncertain text (like "I probably failed the test"), and how you might use a system that predicts factuality in some end task.

https://www.semanticscholar.org/paper/Neural-models-of-factuality-Rudinger-White/4d62a1e7819f9e3f8c837832c66659db5a6d9b37

68 - Neural models of factuality, with Rachel Rudinger

Blog post by Alex Irpan titled "Deep Reinforcement Learning Doesn't Work Yet"
https://www.alexirpan.com/2018/02/14/rl-hard.html

In this episode, Alex Irpan talks about limitations of current deep reinforcement learning methods and why we have a long way to go before they go mainstream. We discuss sample inefficiency, instability, the difficulty to design reward functions and overfitting to the environment. Alex concludes with a list of recommendations he found useful when training models with deep reinforcement learning.

74 - Deep Reinforcement Learning Doesn't Work Yet, with Alex Irpan

In this episode, we invite Hal Daumé to continue the discussion on reinforcement learning, focusing on how it has been used in NLP. We discuss how to reduce NLP problems into the reinforcement learning framework, and circumstances where it may or may not be useful. We discuss imitation learning, roll-in and roll-out, and how to approximate an expert with a reference policy. 

DAgger: https://www.semanticscholar.org/paper/A-Reduction-of-Imitation-Learning-and-Structured-to-Ross-Gordon/17eddf33b513ae1134abadab728bdbf6abab2a05?navId=citing-papers
RESLOPE: http://legacydirs.umiacs.umd.edu/~hal/docs/daume18reslope.pdf

75 - Reinforcement / Imitation Learning in NLP, with Hal Daumé III

In this episode, we invite John Hewitt to discuss his take on how to probe word embeddings for syntactic information. The basic idea is to project word embeddings to a vector space where the L2 distance between a pair of words in a sentence approximates the number of hops between them in the dependency tree. The proposed method shows that ELMo and BERT representations, trained with no syntactic supervision, embed many of the unlabeled, undirected dependency attachments between words in the same sentence.

Paper: https://nlp.stanford.edu/pubs/hewitt2019structural.pdf
GitHub repository: https://github.com/john-hewitt/structural-probes
Blog post: https://nlp.stanford.edu/~johnhew/structural-probe.html
Twitter thread: https://twitter.com/johnhewtt/status/1114252302141886464
John's homepage: https://nlp.stanford.edu/~johnhew/

88 - A Structural Probe for Finding Syntax in Word Representations, with John Hewitt

In this episode, we invite Zhou Yu to give an overview of dialogue systems. We discuss different types of dialogue systems (task-oriented vs. non-task-oriented), the main building blocks and how they relate to other research areas in NLP, how to transfer models across domains, and the different ways used to evaluate these systems. Zhou also shares her thoughts on exciting future directions such as developing dialogue methods for non-cooperative environments (e.g., to negotiate prices) and multimodal dialogue systems (e.g., using video as well as audio/text).

Zhou Yu's homepage: http://zhouyu.cs.ucdavis.edu/

89 - Dialog Systems, with Zhou Yu

In this episode, we invite Jonathan Berant to talk about executable semantic parsing. We discuss what executable semantic parsing is and how it differs from related tasks such as semantic dependency parsing and abstract meaning representation (AMR) parsing. We talk about the main components of a semantic parser, how the formal language affects design choices in the parser, and end with a discussion of some exciting open problems in this space.

Jonathan Berant's homepage: http://www.cs.tau.ac.il/~joberant/

91 - (Executable) Semantic Parsing, with Jonathan Berant

In this episode, we invite David Bamman to give an overview of computational humanities. We discuss examples of questions studied in computational humanities (e.g., characterizing fictionality, assessing novelty, measuring the attention given to male vs. female characters in the literature). We talk about the role NLP plays in addressing these questions and how the accuracy and biases of NLP models can influence the results. We also discuss understudied NLP tasks which can help us answer more questions in this domain such as literary scene coreference resolution and constructing a map of literature geography.

David Bamman's homepage: http://people.ischool.berkeley.edu/~dbamman/
LitBank dataset: https://github.com/dbamman/litbank

92 - Computational Humanities, with David Bamman

**The podcast is currently on hiatus. For more active NLP content, check out the Holistic Intelligence Podcast linked below.**

Welcome to the NLP highlights podcast, where we invite researchers to talk about their work in various areas in natural language processing. All views expressed belong to the hosts/guests, and do not represent their employers.