ML Systems challenges; Multilingual Coref; Unsupervised NMT; Human MT performance; Neural Lattice Language Models; Norms comparison; Are GANs created equal
ML Systems challenges; Multilingual Coref; Unsupervised NMT; Human MT performance; Neural Lattice Language Models; Norms comparison; Are GANs created equal
Enjoy your morning coffee ☕️ with the thought-provoking posts in this issue of NLP News: Listen to Michael Jordan outline challenges for applying ML to systems; train your own coreference resolution model in any language; train an Unsupervised Neural Machine Translation model; discover how Microsoft matched human performance on Machine Translation; learn about Neural Lattice Language Models; and many more.
This edition also features a new category, Best Visualization. Do you find this helpful 💡or boring 💤? Let me know!
Presentations and slides
Tools and implementations
State-of-the-art multilingual coreference resolution
Coreference resolution is important for many NLP tasks but systems exist mainly for English and Chinese because models are complex, open-source implementations are rare, and datasets are almost inexistent. Huggingface tries to change that by open-sourcing a simple neural net model that can be easily trained, modified and used with spaCy and the Prodigy annotator. Also check out the online demo and the associated blog post.
Unsupervised Neural Machine Translation
Unsupervised Neural Machine Translation promises to have a large impact on Machine Translation. Artetxe et al. now release their model Unsupervised NMT system published at ICLR 2018, the first open-source release of such a model.
"Are GANs Created Equal?" code
Training GANs is still challenging. The code for "Are GANs Created Equal? A Large-Scale Study" is a great starting point to apply GANs to new problems, such as natural language generation.
Best visualization
Blog posts and other articles
CRF Layer on the Top of BiLSTM — createmomo.github.io
A Conditional Random Field (CRF) layer is often used as the output layer of BiLSTMs for sequence labelling tasks where labels depend on each other, e.g. Named Entity Recognition (NER). This blog post gives an intuition why this is useful.
Kipoi: Model zoo for genomics — kipoi.org
Molecural biology is an important application area for Machine Learning. As an outsider, it is hard to get started, however. Kipoi (pronounced: kípi) facilitates this by providing an API and a repository of ready-to-use trained models for regulatory genomics, containing 1709 different models.
2018 NLP Challenge on Machine Reading Comprehension
Are you interested in Machine Reading Comprehension? The 2018 NLP Challenge on Machine Reading Comprehension provides a large-scale, open-domain Chinese MRC dataset consisting of 300k questions sampled from real anonymized user queries from Baidu Search. It aims to promote the development of reading comprehension technology. Registration closes on March 30.
Learning to Represent Words by how They’re Spelled — mlatgt.blog
Yuval Pinter discusses how word embeddings can be learned for unseen words based on their spelling, as described in his EMNLP 2017 paper.
AI researchers are always in need of compute and cloud providers such as AWS and GCP make a lot of money renting GPUs on their platforms. If you have GPUs, why not rent them out to researchers in need? You can make ~2x more than mining cryptocurrencies! There are obvious security risks, but whoever manages to solve them, this is an Airbnb for GPUs waiting to happen.
What is wrong with VAEs? — akosiorek.github.io
A blog post with an intuitive structure and build-up that discusses the deficiencies of Variational Autoencoders (VAEs).
Industry insights
Microsoft reaches a historic milestone, using AI to match human performance in translating news from Chinese to English Microsoft reportedly matches human translation performance using several recently proposed improvements to NMT: Transformer (NIPS 2017) + Dual Learning (NIPS 2016) + Deliberation Nets (NIPS 2017). Of course, many of the usual caveats apply. You can read the paper here. Of course, we are still a long way from matching human performance on all texts.
An Introduction to Snips NLU, the Open Source Library behind Snips Embedded Voice Platform — medium.com
Snips open-sources their Natural Language Understanding (NLU) library that enables you to identify a user's intent with high accuracy and low memory footprint.
Allen Institute CEO says A.I. graduates should take oath — www.cnbc.com
The CEO of the Allen Institute for Artificial Intelligence (AI2), Oren Etzioni, argues that graduates in ML should take doctors' Hippocratic oath, tailored to technology. An excerpt:
I must not play at God nor let my technology do so.
Muse, an AI-powered parenting bot, wants to help parents help their children succeed — qz.com
Muse asks parents one question every day, e.g. Have you and Lucas visited a library in the past month? Based on the parents' responses along with unstructured data (audio or video conversations, photos) the system aims to create a constantly evolving profile of each child.
Evolution is the New Deep Learning
In a past edition of this newsletter, we have already seen that evolutionary algorithms are competitive with Deep Reinforcement Learning on many tasks reinforcement learning tasks. In this blog post, Sentient outlines new advances using evolutionary algorithms to find language modelling architectures, for multi-task character recognition, and multi-task attribute detection.
Paper picks
Style transfer, the task of automatically transferring the style of one piece of text into another, is a key task in making conversational agents deal with changing circumstances and domains. While there has been recent advances in this task (Shen et al., NIPS 2017), progress has been hampered by the unavailability of large training datasets. This paper introduces the
largest corpus for a particular stylistic transfer (formality) and shows that techniques from Machine Translation can serve as strong baselines for future style transfer work.
Ranking Sentences for Extractive Summarization with Reinforcement Learning (NAACL 2018)
Some recent approaches have used reinforcement learning to generate abstractive summaries. This paper uses reinforcement learning for extractive summarization. Compared to existing approaches, its training maximizes the ROUGE score not only locally of individual sentences but globally of the entire summary.
Neural Lattice Language Models (TACL 2018)
This paper proses a new framework for language modelling, neural lattice language models. These models construct a lattice of possible paths through a sentence and marginalize across this lattice for optimization. The authors show that they can seamlessly integrate linguistic intuitions such as polysemy this way.