NLP News

By Sebastian Ruder

Regular analyses of advances in natural language processing and machine learning.

Regular analyses of advances in natural language processing and machine learning.

By subscribing, you agree with Revue’s Terms of Service and Privacy Policy and understand that NLP News will receive your email address.




PaLM 🌴, DALL-E 2 👨‍🎨, Chinchilla 🐭, Chain-of-thought prompting ⛓💭✍️, Values and Culture in NLP 🏛

The emergence of large pre-trained models has fundamentally changed the face and nature of progress in ML and NLP. The underlying methods have not changed dramatically; neural networks have already been pre-trained more than 15 years ago. However, the recent …


Highlights, new tasks & graph ML in 2021; Safer pre-trained models; Embeddings: Larger ≠ better

ML and NLP Research Highlights of 2021 💡I wrote up some of my research highlights in 2021 in this post. Overall, most of the trends I observed revolved around pre-trained models and their capabilities—how to train them more effectively, how to do few-shot lea…


Pre-training + Massive Multi-tasking, Benchmarking in NLP, EMNLP primer, 🤗 NLP Course, ACL 2021 recap,

Multi-task learning (MTL), training a model on several tasks at once and sharing information is a general method that is fundamental to training neural networks. Rich Caruana's 1997 paper is one of the best introductions to this topic and as relevant today as…


ICML round-up, Open collaboration, CLIP art, Internet augmentation, New GLUE-style benchmarks

Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation Neural generative models, despite their popularity, are known to suffer from some deficiencies, such as a tendency to generate frequent tokens. Popular methods to address this s…


GitHub Copilot, The Perceiver, Beyond the Transformer, Data augmentation, NL augmenter 🦎 → 🐍, Research communication

OpenAI Codex / GitHub CopilotIf you are working with software, then you've probably heard about the release of GitHub Copilot, a coding assistant based on Codex, a GPT language model fine-tuned on code on GitHub (see the paper). As far as I'm aware, this repr…


EACL, ICLR, NAACL papers round-up, Research reality checks, ML on code

Inspired by an xkcd comic, compilations of "typical" papers in different disciplines such as the above have been making the rounds on social media over the last couple of days. My favourite ones cover machine learning, economy, and neuroscience.Looking beyond…


ICLR 2021 Outstanding Papers, Char Wars, Speech-first NLP, Virtual conference ideas

The best papers of a major ML conference such as ICLR 2021 are often a good indication of new trends in the field. Here's a round-up of the ICLR 2021 award winners (in alphabetical order):Beyond Fully-Connected Layers with Quaternions: Parameterization of Hyp…


QA—how did we get here? 🙋‍♀️, Adapting to time ⏳, Data detectives 🕵️‍♀️

A journey of shooting for the moon—and answering questions about moons and moonshots in the process.Writing this section, I realised that there is already a lot of high-quality information on the topic available. My favourite resource: The Open-Domain QA Tuto…