NLP News - Recurrent Highway Hypernetworks, New Multimodal Environments, LDA2vec, DL for Structured Data, NIPS highlights, AlphaZero, QAngaroo
NLP News - Recurrent Highway Hypernetworks, New Multimodal Environments, LDA2vec, DL for Structured Data, NIPS highlights, AlphaZero, QAngaroo
Some highlights of this newsletter: An implementation of recurrent highway hypernetworks; new multimodal environments for visual question answering; why the intelligence explosion is impossible; a tutorial on LDA2vec; Deep Learning for structured data; lots of highlights from NIPS including tutorial slides, Ali Rahimi's presentation, debate and conversation notes, competition winners; AlphaZero; QAngaroo, a new generation of QA datasets.
When is the last time your teeth felt like they had little sweaters on them?
- A question by an Amazon Mechanical Turk worker to determine if they were talking to a human or AI (credit: Sarah Schwettmann).
Tools and implementations
mordecai - Custom-built full text and event geoparsing — github.com
Many applications require geographic information, but extracting it from text is difficult. mordecai is a Python library for full text geoparsing that extracts the place names from a piece of text, resolves them to the correct place, and returns their coordinates and structured geographic information.
The fast.ai deep learning library, lessons, and tutorials — github.com
fast.ai's deep learning library is known to many from its course. It is now available via pip.
Recurrent Highway Hypernetworks — github.com
Pytorch code for Character-level Language Modeling with Recurrent Highway Hypernetworks (NIPS 2017).
A Household multimodal environment — github.com
HoME (Household Multimodal Environment) is a platform for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context.
Realistic and Rich 3D Environments for Embodied Question Answering — embodiedqa.org
Staying with multimodal environments, Embodied Question Answering (QA) combines Visual QA and robot navigation. In order to answer a question, the agent must first explore the environment.
pycolab game engine — github.com
We are not done with a environments yet. pycolab is a highly-customisable gridworld game engine with some batteries included that allows you to create your own gridworld games to test reinforcement learning agents.
Unsupervised Language Modeling at scale for robust sentiment classification — github.com
Implementation of Generating Reviews and Discovering Sentiment by OpenAI that reduces training time from 1 month to 5 days.
Reports on benchmarks
The 2017 Artificial Intelligence Annual Report gives a comprehensive overview of the role of AI in industry, academia, and sheds light on current technical performance.
Machine learning benchmarks: Hardware providers — rare-technologies.com
A comparison of machine learning frameworks and hardware platforms from practical aspects, such as their ease of use, cost, stability, scalability and performance.
Chatbots
Your chatbot needs a personality — blog.myralabs.com This blog post emphasizes that the current generation of chatbots require a personality to succeed. But maybe this is the wrong way to think about it, as pointed out by a HN user:
We do not yet have chatbots. What we do have are publicly-facing undocumented nondeterministic command line interfaces which expect the user to guess the right commands. This interface further insults the user by pretending to be a person.
‘The Woebot will see you now’ — the rise of chatbot therapy
A Washington post article about the dozens of online apps that offer instant therapy for the lonely, the confused and the anxious via chatbots.
Alexa Prize winners — developer.amazon.com
The winners of the Alexa Prize 2017 have been announced. The winning system by University of Washington was able to sustain conversations with an average duration of 10 minutes, 22 seconds and an average score of 3.17/5.
The promise and danger of AI
Using Artificial Intelligence to Augment Human Intelligence — distill.pub
A great article that outlines directions of how we can augment our intellect using AI, in line with early ideas of human-computer symbiosis envisioned by pioneers such as J.C.R. Licklider, Douglas Engelbart, and Alan Kay.
The impossibility of intelligence explosion — medium.com
An essay by François Chollet on why the singularity, i.e. the explosion of super-human intelligence is impossible.
Specifying AI safety problems in simple environments — deepmind.com
This blog posts discusses a recent DeepMind paper on AI safety that introduces a selection of simple reinforcement learning environments designed specifically to measure ‘safe behaviours’.
Bias
Interpreting Deep Neural Networks with SVCCA — research.googleblog.com
This blog post discusses SVCCA, a novel simple and scalable method to make models more interpretable.
Debugging data: Microsoft researchers look at ways to train AI systems to reflect the real world — blogs.microsoft.com
A blog post looking at the work of Microsoft researchers such as Hanna Wallach, Kate Crawford, and Patrice Simard to reduce bias in ML models.
Tutorials
LDA2vec: Word Embeddings in Topic Models — towardsdatascience.com
Lars Hulstaert gives a great overview of LDA2vec, a model that learns dense word vectors jointly with Dirichlet-distributed latent document-level mixtures of topic vectors.
Sequence Modeling with CTC — distill.pub A visual guide to Connectionist Temporal Classification, an algorithm used to train deep neural networks in speech recognition, handwriting recognition and other sequence problems.
Reproducible Data Analysis in Jupyter
A series of tutorials on conducting reproducible data analysis in Jupyter notebooks.
More blog posts and articles
1st place in Porto Seguro’s Safe Driver Prediction with representation learning
The 1st place winner of the Safe Driver Prediction kaggle competition describes his approach: A rare competition with structured data where XGBoost did not yield victory. Instead, neural networks the winner trained neural networks on representations learned with a denoising autoencoder.
AI•ON: Artificial Intelligence Open Network — ai-on.org
AI•ON, the Artificial Intelligence Open Network is relaunching with project proposals from members of the Google Brain, Berkeley, Princeton, MILA, and the University of Amsterdam.
Search ICLR 2018: Top 100 papers — search.iclr2018.smerity.com
The clear interface to search ICLR 2018 by Stephen Merity now allows to see the review scores and the top 100 papers.
Artificial intelligence goes bilingual—without a dictionary — www.sciencemag.org
An article in Science Magazine about the two recent papers on unsupervised Neural Machine Translation.
Recurrent Relational Networks — rasmusbergpalm.github.io
A blog post introducing Recurrent Relational Networks, a recurrent extension of Relational Networks (NIPS 2017).
NIPS 2017
Deep Probabilistic Modelling with Gaussian Processes
Slides of the tutorial on Deep Probabilistic Modelling with Gaussian Processes by Neil D. Lawrence.
Geometric deep learning on graphs and manifolds tutorial — www.dropbox.com
Slides of the tutorial on geometric deep learning on graphs and manifolds.
Accepted papers of the Machine Learning for Creativity and Design workhshop — nips2017creativity.github.io
Accepted papers of the Machine Learning for Creativity and Design workshop. The AI-created art can be seen here.
Safe and Nested Subgame Solving for Imperfect-Information Games
One of the NIPS 2017 best papers, which introduces techniques for solving imperfect-information games that were used in Libratus, the first AI that defeated top humans in heads-up no-limit Texas hold'em poker. A teaser of the paper can be seen here.
Ali Rahimi's Test-of-time award presentation — www.youtube.com
In his presentation, Ali Rahimi likens the current generation of ML models to alchemy, which the community has failed to underpin with theory. Yann LeCun rebuts his arguments, to which Rahimi replies here.
Machine Learning for Systems and Systems for Machine Learning
Jeff Dean's presentation on using ML to improve systems, e.g. compilers, device placement, etc. and creating new systems for ML, e.g. TPUs, linear algebra libraries, etc.
Tesla's conversation on AI with Andrej Karpathy, Elon Musk, and others
Stephen Merity chronicles Tesla's conversation on AI, which touches on Tesla's hardware and software for AI, self-driving cars, AGI, and other topics.
Dave Gershgorn documents the debate on interpretability with Rich Caruana, Patrice Simard, Kilian Weinberger, and Yann LeCun.
The white paper of the winning system at the NIPS 2017 Human-Computer QA competition.
Industry insights
Facebook AI Research Residency Program — research.fb.com
Facebook AI Research has announced a one-year research training program designed to give you hands-on experience of machine learning research.
Victims of Sexual Harassment Have a New Resource: AI — www.technologyreview.com
Montreal-based Botler AI launched a system that provides free information and guidance to those who have been sexually harassed and are unsure of their legal rights.
How Reuters’s Revolutionary AI System Gathers Global News — www.technologyreview.com
Reuters ML system uses Twitter to record news events as they as they are happening.
Facebook to expand artificial intelligence to help prevent suicide — www.reuters.com
Facebook Inc will expand its ML software to detect users with suicidal intent to other countries after successful tests in the US.
Paper picks
Are GANs Created Equal? A Large-Scale Study
With the GAN zoo growing and growing, there is no clear evidence which is the best GAN. This large-scale study evaluates the most popular GAN variants and -- controversially -- finds no evidence that any of the tested algorithms consistently outperforms the original one.
Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples
In the line of research on interpreting neural networks, one direction is to model the underlying complex model with a simpler model. This paper proposes a method to extract an automaton from an LSTM. The technique can highlight cases where the LSTM fails to learn the intended generalization, as the extracted automata is overly complex compared to a simple target language.
Ethical Challenges in Data-driven Dialogue Systems
Chatbots and dialogue systems are becoming more commonplace, but can lead to biased or offensive conversations. This study highlights potential ethical issues that arise in dialogue systems research, e.g. implicit biases, adversarial examples, privacy violations, safety concerns, RL considerations, and reproducibility concerns.
Neural Text Generation: A Practical Guide
This paper gives an overview of encoder-decoder architectures for neural text generation and highlights some best practices.
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
The most recent article discussing the latest generation of AlphaGo, Alpha Zero. Similar to AlphaGo Zero, Alpha Zero has learned chess and shogi entirely from scratch using self-play. Alpha Zero's games defeating Stockfish, the strongest open-source chess engine in the world, are well worth watching.
Dataset spotlight
QAngaroo - Reading Comprehension with Multiple Hops — qangaroo.cs.ucl.ac.uk
Question answering and reading comprehension has received a large amount of interest in recent years fuelled by new and large datasets such as CNN News, SQuAD, and MSMARCO. All of these datasets, however, simplify the task in that the provided passage is known to contain the answer. Welbl et al. from UCL introduce QAngaroo, a collection of two new datasets that model the more realistic setting of first having to find the relevant passage. These datasets should drive the development of QA systems and should make them more useful for more realistic applications.