Fake video, State of ML Code, Notebooks, and Software, Advice for MScing and PhDing, Ada Lovelace, 150+ Best Resources

In the past two weeks, not one but two new approaches for generating increasingly photo-realistic videos have been proposed. The first one from NVIDIA generates a video based on semantic labels. The second one from UC Berkeley transfers the motion of a person in a source video to a target person. Both have video demonstrations available that are fun to watch (here and here).

Tell me more The nice (and somewhat contrarian) thing about the Berkeley approach is that it not done end-to-end, but uses detected poses as an intermediate representation to anchor the final output.

Why is this important? Fake videos are nothing new, but so far, they were still expensive to produce or easy to spot (see last year’s Deepfakes). These new approaches indicate that we may soon cross the threshold where it may be hard for non-experts to tell the difference between what is generated and what is fake. In addition to systems for detecting and flagging fake news, we will thus need better mechanisms that identify and flag fake video content as part of our social media platforms.

Code Python is the de facto lingua franca of Machine Learning. There are few people more suitable to comment on its current state than Travis Oliphant, creator of numpy and scipy and founder of Anaconda. TL;DR: Discussing the current ML frameworks, he highlights that most libraries chose to reimplement numpy and scipy as needed. Pre 2016, the Python scientific ecosystem was mainly shaped by a few people. Numpy was created to unify array objects in Python and to unify the PyData community. While it has now evolved into a large community effort, the large number of DL frameworks essentially “divide” the community again.

Notebooks IPython notebooks are a popular tool and the main development environment for many data scientists. AI2’s Joel Grus gives a contrarian and meme-filled presentation on what not to like about notebooks, focusing mainly on their hidden state and arbitrary execution order, which makes reasoning about state difficult. Come for the memes, stay for the engineering rigour.

Hardware Tim Dettmers has updated his widely read post on which GPUs to get for Deep Learning. TL;DR: NVIDIA is still the best bet. If you want to run RNNs on GPUs, bandwith is the most important performance metric, while for convolutional neural networks, it is FLOPS. For the typical Deep Learning practicioner and researcher, an RTX or a cheap GTX 1080/1070/1060 is the best choice.

Getting started with research takes practice and even seasoned researchers make mistakes. The hallmark of a good researcher (similar to a good coder) is not to write flawless code, but to know how to efficiently probe a model and identify bugs. Tim Rocktäschel, Jakob Foerster, and Greg Farquhar give some great advice along these lines on what to consider when working on (short-term) ML projects.

While many places still require a PhD to do research, as ML starts to permeate every business area, a PhD is becoming less crucial. Doing a PhD is a decision that should be carefully considered. To this end, fast.ai’s Rachel provides some useful advice highlighting some of the downsides that a PhD entails.

Designing a good poster for a conference, similar to designing a good presentation, takes practice. Xanda Schofield has some advice on how to design a poster faster and better—by focusing on the presentation of the poster first and foremost and bootstrapping the rest of the poster from the presentation.

While the discourse around AI generally revolves around future capabilities of AI, it is often productive to look back and take inspiration from the humble beginnings. It is valuable to look back not only on the beginnings of AI, but also the beginnings of programming in general. Ada Lovelace is generally thought to be the first programmer, but the technical details of her program are typically not given much attention. This article superbly contextualizes the program by discussing number theory, the Bernoulli numbers, and Charles’ Babbage’s Analytical Engine. Referred to as “Lady Lovelace’s Objection” by Alan Turing, Ada Lovelace argued that despite the Analytical Machine’s capabilities, it cannot really be said to “think”, marking perhaps the first consideration of Artificial Intelligence.

Looking at the more immediate past, last week OpenAI lost 0-2 against two top Dota 2 teams at The International, the biggest annual Dota 2 tournament. This Verge article gives a nuanced take of what this loss means and eloquently ties it to the broader conversation of what human-AI competitions allow us to measure and whether human-AI competitions are a good way to benchmark future progress in AI.

Deep Learning to Learn 🤖 Pieter Abbeel discusses recent meta-learning approaches to enable RL models to learn faster, bridging the gap to human learning.

Planting the Seeds of Probabilistic Thinking 🌱 Shakir Mohamed gives an overview of valuable foundations and tricks for probabilistic thinking and then delves into some of the algorithms, focusing on deep generative models.

150+ of the Best Machine Learning, NLP, and Python Tutorials (Updated) 📚 An updated list of many of the best resources available to learn about ML, NLP, and Python.

I want to learn Artificial Intelligence and Machine learning. Where can I start? 🤖 Daniel Bourke gives an inspiring account of his road to becoming an ML engineer and the resources he used for learning in this Quora answer.

Dopamine 👾 A new Tensorflow-based framework for fast prototyping of reinforcement learning algorithms. See this Google AI blog post for more information.

TransmogrifAI 🕴 An end-to-end automated ML library for structured data used in production by Salesforce’s Einstein AI. See this Salesforce blog post for more information.

NLP’s generalization problem, and how researchers are tackling it 🔬 Ana Marasović highlights ways in which state-of-the-art systems for many NLP tasks are still brittle and spurious and discusses the three main directions (more inductive biases, common sense, and evaluation on unseen tasks and distributions) in which researchers are tackling the problem.

Simple guide to Neural Arithmetic Logic Units 👩‍🔬 Faizan Patel gives a nice overview and explanation of the intuition behind DeepMind’s recent paper Neural Arithmetic Logic Units.

Moving Beyond Translation with the Universal Transformer 🔃 This Google AI blog post gives an overview of the recent Universal Transformer model, an extension of the Transformer with adaptive computation time.

ICML+ACL’18: Structure Back in Play, Translation Wants More Context 🏛 André Martins, VP of AI Research at Unbabel shares his extensive thoughts on ICML and ACL 2018, focusing on structured prediction, deep generative models, seq2seq models, and representation learning at ICML, and machine translation, generation, and parsing at ACL.

Recommender Systems: Exploring the Unknown Using Uncertainty 🔦 Inbar Naor Yoel Zeldes and Inbar Naor describe the exploration-exploitation problem in recommender systems and various methods to deal with this challenge.

Unsupervised machine translation: A novel approach to provide fast, accurate translations for more languages 🇬🇧➡️🇩🇪 This blog post based on an EMNLP 2018 paper from Facebook is a nice overview of the current state-of-the-art in unsupervised MT.

Adversarial Removal of Demographic Attributes from Text Data (EMNLP 2018) Elazar and Goldberg show that representations of text classifiers can be used to recover demographic information. The decisions of classifiers are thus not agnostic to demographic attributes. Even if the models are trained with an adversarial component, a post-hoc classifier trained on the encoded sentences can still predict demographic attributes with reasonable accuracy. We should thus not rely on adversarial training to obfuscate sensitive information.

🦆QuAC : Question Answering in Context (EMNLP 2018) and CoQA: A Conversational Question Answering Challenge (arXiv) Two datasets on conversational question answering have been recently proposed. In both cases, two crowd workers are paired up as a teacher and a student: The student asks free-text questions and the teacher provides answers based on a text passage. There are several differences between the datasets:

Coreference and pragmatic reasoning is relevant for both and state-of-the-art reading comprehension systems trained on the datasets fall short of human performance by about 20%, indicating a lot of potential for future work.