NLP News
By Sebastian Ruder
issues
PaLM 🌴, DALL-E 2 👨🎨, Chinchilla 🐭, Chain-of-thought prompting ⛓💭✍️, Values and Culture in NLP 🏛
The emergence of large pre-trained models has fundamentally changed the face and nature of progress in ML and NLP. The underlying methods have not changed dramatically; neural networks have already been pre-trained more than 15 years ago. However, the recent …
Highlights, new tasks & graph ML in 2021; Safer pre-trained models; Embeddings: Larger ≠ better
ML and NLP Research Highlights of 2021 💡I wrote up some of my research highlights in 2021 in this post. Overall, most of the trends I observed revolved around pre-trained models and their capabilities—how to train them more effectively, how to do few-shot lea…
Pre-training + Massive Multi-tasking, Benchmarking in NLP, EMNLP primer, 🤗 NLP Course, ACL 2021 recap,
Multi-task learning (MTL), training a model on several tasks at once and sharing information is a general method that is fundamental to training neural networks. Rich Caruana's 1997 paper is one of the best introductions to this topic and as relevant today as…
ICML round-up, Open collaboration, CLIP art, Internet augmentation, New GLUE-style benchmarks
Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation Neural generative models, despite their popularity, are known to suffer from some deficiencies, such as a tendency to generate frequent tokens. Popular methods to address this s…
GitHub Copilot, The Perceiver, Beyond the Transformer, Data augmentation, NL augmenter 🦎 → 🐍, Research communication
OpenAI Codex / GitHub CopilotIf you are working with software, then you've probably heard about the release of GitHub Copilot, a coding assistant based on Codex, a GPT language model fine-tuned on code on GitHub (see the paper). As far as I'm aware, this repr…
EACL, ICLR, NAACL papers round-up, Research reality checks, ML on code
Inspired by an xkcd comic, compilations of "typical" papers in different disciplines such as the above have been making the rounds on social media over the last couple of days. My favourite ones cover machine learning, economy, and neuroscience.Looking beyond…
ICLR 2021 Outstanding Papers, Char Wars, Speech-first NLP, Virtual conference ideas
The best papers of a major ML conference such as ICLR 2021 are often a good indication of new trends in the field. Here's a round-up of the ICLR 2021 award winners (in alphabetical order):Beyond Fully-Connected Layers with Quaternions: Parameterization of Hyp…
QA—how did we get here? 🙋♀️, Adapting to time ⏳, Data detectives 🕵️♀️
A journey of shooting for the moon—and answering questions about moons and moonshots in the process.Writing this section, I realised that there is already a lot of high-quality information on the topic available. My favourite resource: The Open-Domain QA Tuto…
IE—how did we get here?, Large LMs, The human side of ML
This recent xkcd comic struck a chord with me. So much more can be done with language beyond treating it as data for training ML models.
EMNLP 2020 recap, Minimum viable datasets, Efficiency, Roguelikes for RL Research
EMNLP 2020 took place last month. Overall, the interactions in the virtual conference venue made it feel closer to an actual conference for me. I particularly enjoyed a number of serendipitous encounters in the corridors while trying to navigate the conferenc…
EMNLP 2020 & ICLR 2021, Toxicity detection, Data augmentation and adversarial examples
The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) will take place from November 16–18 (with workshops and tutorials from November 19–20). Registration is still open ($200 regular and $75 for students). The schedule and proce…
ML and NLP starter toolkit, Low-resource NLP toolkit, "Can a LM understand natural language?", The next generation of NLP benchmarks
I am amazed every day how dramatically the field has changed over the last years and how many expertly created resources are now out there to help you get started in the field. Here are my favourite resources to kick-start your learning journey:The NLP Pandec…
Reviewing, Taking stock, Theme papers, Poisoning and stealing models, Multimodal generation
With the EMNLP and NeurIPS deadlines almost behind us, the topic of reviewing is of course being discussed again. In particular, the EMNLP Organization team published some great advice on the EMNLP blog. The post gives clear advice on what are often invalid b…
COVID-19 edition 😷
Self-isolation can be hard on each of us—but have you considered the impact it might have on our models?
COVID-19, Hutter Prize, Compression = AGI?, BERT, Green AI
Updates during the last month include:New results on English-Hindi machine translationNew results on intent detection (with code)State-of-the-art results on CNN / Daily Mail summarizationState-of-the-art results on coreference resolutionState-of-the-art langu…
Accelerating science, memorizing vs learning to look things up, Schmidhuber's 2010s, Greek BERT, ARC, Illustrated Reformer, Annotated GPT-2, oLMpics
Updates during the last month include:Code for the Mogrifier LSTM (ICLR 2020), SOTA on PTB and WikiText-2 is now onlineNew SOTA models for text simplificationNew dataset on Gendered Ambiguous Pronoun (GAP) resolutionResults from the Dialogue System Technology…
NLP Progress, Restrospectives and look ahead, New NLP courses, Independent research initiatives, Interviews, Lots of resources
In order to make tracking new results in NLP easier, I'll be compiling the most exciting additions to NLP-progress in a monthly release (thanks to Reza for the suggestion):Updates during the last month include:A new state of the art summarisation datasetsNew …
2020 NLP wish lists, HuggingFace + fastai, NeurIPS 2019, GPT-2 things, Machine Learning Interviews
My wish list for NLP research in 2020:Learning from few samples rather than from large datasetsCompact and efficient rather than huge modelsEvaluate on at least another language (from a different language family)New datasets contain at least one other languag…
Highlights of EMNLP 2019, Ethics in NLP vol. 2, AI and Journalism
The compute needed for training state-of-the-art ML models has increased tremendously over the last years. To get a feeling for exactly how dramatic this increase has been, it helps to contrast this with the historical growth of compute in AI (see below). In …
Deep Learning Indaba, EurNLP, ML echo chamber, Pretrained LMs, Reproducibility papers
ICLR 2020 submissions 📑 In contrast to many other ML and NLP venues, ICLR 2020 submissions are available from the date they are submitted. In light of this onslaught of papers (2,600 this year), don't forget to keep a cool head and continue to pursue your own…