current

NIPS 2017 paper.

We dig into the details of the Transformer, from the "attention is all you need" paper.  Ashish and Jakob give us some motivation for replacing RNNs and CNNs with a more parallelizable self-attention mechanism, they describe how this mechanism works, and then we spend the bulk of the episode trying to get their intuitions for _why_ it works.  We discuss the positional encoding mechanism, multi-headed attention, trying to use these ideas to replace encoders in other models, and what the self-attention actually learns.  Turns out that the lower layers learn something like n-grams (similar to CNNs), and the higher layers learn more semantic-y things, like coreference.

https://www.semanticscholar.org/paper/Attention-Is-All-You-Need-Vaswani-Shazeer/0737da0767d77606169cbf4187b83e1ab62f6077

Minor correction: Talking about complexity equations without the paper in front of you can be tricky, and Ashish and Jakob may have gotten some of the details slightly wrong when we were discussing computational complexity.  The high-level point is that self-attention is cheaper than RNNs when the hidden dimension is higher than the sequence length.  See the paper for more details.

36 - Attention Is All You Need, with Ashish Vaswani and Jakob Uszkoreit

EMNLP 2017 paper by Ramakanth Pasunuru and Mohit Bansal

Ram and Mohit join us to talk about their work, which uses reinforcement learning to improve performance on a video captioning task.  They directly optimize CIDEr, a popular image/video captioning metric, using policy gradient methods, then use a modified version of CIDEr that penalizes the model when it fails to produce a caption that is _entailed_ by the correct caption.  In our discussion, we hit on what video captioning is, what typical models look like for this task, and how the entailment-based reward function is similar to other attempts to be smart about handling paraphrases when evaluating or training language generation models.

Unfortunately, due to some technical issues, the audio recording is a little worse than usual for this episode.  Our apologies.

https://www.semanticscholar.org/paper/Reinforced-Video-Captioning-with-Entailment-Reward-Pasunuru-Bansal/0d11977afa1a6ce90dc3b1f26694492c2ab04773

current

Sabarish Sankaranarayanan