issues
The emergence of large pre-trained models has fundamentally changed the face and nature of progress in ML and NLP. The underlying methods have not changed dramatically; neural networks have already been pre-trained more than 15 years ago. However, the recent …
ML and NLP Research Highlights of 2021 💡I wrote up some of my research highlights in 2021 in this post. Overall, most of the trends I observed revolved around pre-trained models and their capabilities—how to train them more effectively, how to do few-shot lea…
Multi-task learning (MTL), training a model on several tasks at once and sharing information is a general method that is fundamental to training neural networks. Rich Caruana's 1997 paper is one of the best introductions to this topic and as relevant today as…
Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation Neural generative models, despite their popularity, are known to suffer from some deficiencies, such as a tendency to generate frequent tokens. Popular methods to address this s…
OpenAI Codex / GitHub CopilotIf you are working with software, then you've probably heard about the release of GitHub Copilot, a coding assistant based on Codex, a GPT language model fine-tuned on code on GitHub (see the paper). As far as I'm aware, this repr…
Inspired by an xkcd comic, compilations of "typical" papers in different disciplines such as the above have been making the rounds on social media over the last couple of days. My favourite ones cover machine learning, economy, and neuroscience.Looking beyond…
The best papers of a major ML conference such as ICLR 2021 are often a good indication of new trends in the field. Here's a round-up of the ICLR 2021 award winners (in alphabetical order):Beyond Fully-Connected Layers with Quaternions: Parameterization of Hyp…
A journey of shooting for the moon—and answering questions about moons and moonshots in the process.Writing this section, I realised that there is already a lot of high-quality information on the topic available. My favourite resource: The Open-Domain QA Tuto…
This recent xkcd comic struck a chord with me. So much more can be done with language beyond treating it as data for training ML models.
EMNLP 2020 took place last month. Overall, the interactions in the virtual conference venue made it feel closer to an actual conference for me. I particularly enjoyed a number of serendipitous encounters in the corridors while trying to navigate the conferenc…
The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) will take place from November 16–18 (with workshops and tutorials from November 19–20). Registration is still open ($200 regular and $75 for students). The schedule and proce…
I am amazed every day how dramatically the field has changed over the last years and how many expertly created resources are now out there to help you get started in the field. Here are my favourite resources to kick-start your learning journey:The NLP Pandec…
With the EMNLP and NeurIPS deadlines almost behind us, the topic of reviewing is of course being discussed again. In particular, the EMNLP Organization team published some great advice on the EMNLP blog. The post gives clear advice on what are often invalid b…
Self-isolation can be hard on each of us—but have you considered the impact it might have on our models?
Updates during the last month include:New results on English-Hindi machine translationNew results on intent detection (with code)State-of-the-art results on CNN / Daily Mail summarizationState-of-the-art results on coreference resolutionState-of-the-art langu…
Updates during the last month include:Code for the Mogrifier LSTM (ICLR 2020), SOTA on PTB and WikiText-2 is now onlineNew SOTA models for text simplificationNew dataset on Gendered Ambiguous Pronoun (GAP) resolutionResults from the Dialogue System Technology…
In order to make tracking new results in NLP easier, I'll be compiling the most exciting additions to NLP-progress in a monthly release (thanks to Reza for the suggestion):Updates during the last month include:A new state of the art summarisation datasetsNew …
My wish list for NLP research in 2020:Learning from few samples rather than from large datasetsCompact and efficient rather than huge modelsEvaluate on at least another language (from a different language family)New datasets contain at least one other languag…
The compute needed for training state-of-the-art ML models has increased tremendously over the last years. To get a feeling for exactly how dramatic this increase has been, it helps to contrast this with the historical growth of compute in AI (see below). In …
ICLR 2020 submissions 📑 In contrast to many other ML and NLP venues, ICLR 2020 submissions are available from the date they are submitted. In light of this onslaught of papers (2,600 this year), don't forget to keep a cool head and continue to pursue your own…