Philip Bachman

NeurIPS Conference 2023 Conference Paper

Ignorance is Bliss: Robust Control via Information Gating

• Manan Tomar
• Riashat Islam
• Matthew Taylor
• Sergey Levine
• Philip Bachman

Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations. We propose information gating as a way to learn parsimonious representations that identify the minimal information required for a task. When gating information, we can learn to reveal as little information as possible so that a task remains solvable, or hide as little information as possible so that a task becomes unsolvable. We gate information using a differentiable parameterization of the signal-to-noise ratio, which can be applied to arbitrary values in a network, e. g. , erasing pixels at the input layer or activations in some intermediate layer. When gating at the input layer, our models learn which visual cues matter for a given task. When gating intermediate layers, our models learn which activations are needed for subsequent stages of computation. We call our approach InfoGating. We apply InfoGating to various objectives such as multi-step forward and inverse dynamics models, Q-learning, and behavior cloning, highlighting how InfoGating can naturally help in discarding information not relevant for control. Results show that learning to identify and use minimal information can improve generalization in downstream tasks. Policies based on InfoGating are considerably more robust to irrelevant visual features, leading to improved pretraining and finetuning of RL models.

ICLR Conference 2021 Conference Paper

Data-Efficient Reinforcement Learning with Self-Predictive Representations

• Max Schwarzer
• Ankesh Anand
• Rishab Goel
• R. Devon Hjelm
• Aaron C. Courville
• Philip Bachman

While deep reinforcement learning excels at solving tasks where large amounts of data can be collected through virtually unlimited interaction with the environment, learning from limited interaction remains a key challenge. We posit that an agent can learn more efficiently if we augment reward maximization with self-supervised objectives based on structure in its visual input and sequential interaction with the environment. Our method, Self-Predictive Representations (SPR), trains an agent to predict its own latent state representations multiple steps into the future. We compute target representations for future states using an encoder which is an exponential moving average of the agent’s parameters and we make predictions using a learned transition model. On its own, this future prediction objective outperforms prior methods for sample-efficient deep RL from pixels. We further improve performance by adding data augmentation to the future prediction loss, which forces the agent’s representations to be consistent across multiple views of an observation. Our full self-supervised objective, which combines future prediction and data augmentation, achieves a median human-normalized score of 0.415 on Atari in a setting limited to 100k steps of environment interaction, which represents a 55% relative improvement over the previous state-of-the-art. Notably, even in this limited data regime, SPR exceeds expert human scores on 7 out of 26 games. We’ve made the code associated with this work available at https://github.com/mila-iqia/spr.

ICML Conference 2021 Conference Paper

Decomposed Mutual Information Estimation for Contrastive Representation Learning

• Alessandro Sordoni
• Nouha Dziri
• Hannes Schulz
• Geoffrey J. Gordon
• Philip Bachman
• Remi Tachet des Combes

Recent contrastive representation learning methods rely on estimating mutual information (MI) between multiple views of an underlying context. E. g. , we can derive multiple views of a given image by applying data augmentation, or we can split a sequence into views comprising the past and future of some step in the sequence. Contrastive lower bounds on MI are easy to optimize, but have a strong underestimation bias when estimating large amounts of MI. We propose decomposing the full MI estimation problem into a sum of smaller estimation problems by splitting one of the views into progressively more informed subviews and by applying the chain rule on MI between the decomposed views. This expression contains a sum of unconditional and conditional MI terms, each measuring modest chunks of the total MI, which facilitates approximation via contrastive bounds. To maximize the sum, we formulate a contrastive lower bound on the conditional MI which can be approximated efficiently. We refer to our general approach as Decomposed Estimation of Mutual Information (DEMI). We show that DEMI can capture a larger amount of MI than standard non-decomposed contrastive bounds in a synthetic setting, and learns better representations in a vision domain and for dialogue generation.

NeurIPS Conference 2021 Conference Paper

Pretraining Representations for Data-Efficient Reinforcement Learning

• Max Schwarzer
• Nitarshan Rajkumar
• Michael Noukhovitch
• Ankesh Anand
• Laurent Charlin
• R Devon Hjelm
• Philip Bachman
• Aaron C. Courville

Data efficiency is a key challenge for deep reinforcement learning. We address this problem by using unlabeled data to pretrain an encoder which is then finetuned on a small amount of task-specific data. To encourage learning representations which capture diverse aspects of the underlying MDP, we employ a combination of latent dynamics modelling and unsupervised goal-conditioned RL. When limited to 100k steps of interaction on Atari games (equivalent to two hours of human experience), our approach significantly surpasses prior work combining offline representation pretraining with task-specific finetuning, and compares favourably with other pretraining methods that require orders of magnitude more data. Our approach shows particular promise when combined with larger models as well as more diverse, task-aligned observational data -- approaching human-level performance and data-efficiency on Atari in our best setting.

NeurIPS Conference 2020 Conference Paper

Deep Reinforcement and InfoMax Learning

• Bogdan Mazoure
• Remi Tachet des Combes
• Thang Long Doan
• Philip Bachman
• R Devon Hjelm

We posit that a reinforcement learning (RL) agent will perform better when it uses representations that are better at predicting the future, particularly in terms of few-shot learning and domain adaptation. To test that hypothesis, we introduce an objective based on Deep InfoMax (DIM) which trains the agent to predict the future by maximizing the mutual information between its internal representation of successive timesteps. We provide an intuitive analysis of the convergence properties of our approach from the perspective of Markov chain mixing times, and argue that convergence of the lower bound on mutual information is related to the inverse absolute spectral gap of the transition model. We test our approach in several synthetic settings, where it successfully learns representations that are predictive of the future. Finally, we augment C51, a strong distributional RL agent, with our temporal DIM objective and demonstrate on a continual learning task (inspired by Ms. ~PacMan) and on the recently introduced Procgen environment that our approach improves performance, which supports our core hypothesis.

ICLR Conference 2019 Conference Paper

Learning deep representations by mutual information estimation and maximization

• R. Devon Hjelm
• Alex Fedorov
• Samuel Lavoie
• Karan Grewal
• Philip Bachman
• Adam Trischler
• Yoshua Bengio

This work investigates unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and compares favorably with fully-supervised learning on several classification tasks in with some standard architectures. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

NeurIPS Conference 2019 Conference Paper

Learning Representations by Maximizing Mutual Information Across Views

• Philip Bachman
• R Devon Hjelm
• William Buchwalter

We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-temporal context by observing it from different locations (e. g. , camera positions within a scene), and via different modalities (e. g. , tactile, auditory, or visual). Or, an ImageNet image could provide a context from which one produces multiple views by repeatedly applying data augmentation. Maximizing mutual information between features extracted from these views requires capturing information about high-level factors whose influence spans multiple views – e. g. , presence of certain objects or occurrence of certain events. Following our proposed approach, we develop a model which learns image representations that significantly outperform prior methods on the tasks we consider. Most notably, using self-supervised learning, our model learns representations which achieve 68. 1% accuracy on ImageNet using standard linear evaluation. This beats prior results by over 12% and concurrent results by 7%. When we extend our model to use mixture-based representations, segmentation behaviour emerges as a natural side-effect. Our code is available online: https: //github. com/Philip-Bachman/amdim-public.

ICML Conference 2018 Conference Paper

Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data

• Amjad Almahairi
• Sai Rajeswar
• Alessandro Sordoni
• Philip Bachman
• Aaron C. Courville

Learning inter-domain mappings from unpaired data can improve performance in structured prediction tasks, such as image segmentation, by reducing the need for paired data. CycleGAN was recently proposed for this problem, but critically assumes the underlying inter-domain mapping is approximately deterministic and one-to-one. This assumption renders the model ineffective for tasks requiring flexible, many-to-many mappings. We propose a new model, called Augmented CycleGAN, which learns many-to-many mappings between domains. We examine Augmented CycleGAN qualitatively and quantitatively on several image datasets.

AAAI Conference 2018 Conference Paper

Deep Reinforcement Learning That Matters

• Peter Henderson
• Riashat Islam
• Philip Bachman
• Joelle Pineau
• Doina Precup
• David Meger

In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

ICML Conference 2017 Conference Paper

Learning Algorithms for Active Learning

• Philip Bachman
• Alessandro Sordoni
• Adam Trischler

We introduce a model that learns active learning algorithms via metalearning. For a distribution of related tasks, our model jointly learns: a data representation, an item selection heuristic, and a prediction function. Our model uses the item selection heuristic to construct a labeled support set for training the prediction function. Using the Omniglot and MovieLens datasets, we test our model in synthetic and practical settings.

RLDM Conference 2017 Conference Abstract

Learning Algorithms for Active Learning

• Philip Bachman

For many real-world tasks, labeled data is scarce while unlabeled data is abundant. In active learning, a model selects unlabeled instances for labeling so as to maximize a combination of task perfor- mance and data efficiency. Active learning is useful in many real-world scenarios. For example, in cold-start movie recommendation, a system aims to suggest movies to a new user; preference information for this user is initially unavailable, but may be obtained online by asking her to rate a selection of movies. Known ratings could inform the choice of future queries, to better estimate the user’s preferences with fewer queries overall. Or consider medical image classification, where labeling images is costly because it requires a specialist. Labeling costs could be reduced by clever strategies for selecting images to label. In contrast to most prior work on active learning, which relies on carefully designed heuristics for selecting instances to label, we propose learning active learning algorithms end-to-end via metalearning. I. e. , we propose a model which learns a selection heuristic, and how to use it, by interacting with data from many related tasks. Our model builds on methods developed for reinforcement and one-shot learning. Across a collection of prob- lems based on the Omniglot dataset, our model performs well relative to a set of strong baselines. We show that our model offers promising performance in a practical setting using the MovieLens dataset to simulate the cold-start problem faced by recommendation systems.

NeurIPS Conference 2016 Conference Paper

An Architecture for Deep, Hierarchical Generative Models

• Philip Bachman

We present an architecture which lets us train deep, directed generative models with many layers of latent variables. We include deterministic paths between all latent variables and the generated output, and provide a richer set of connections between computations for inference and generation, which enables more effective communication of information throughout the model during training. To improve performance on natural images, we incorporate a lightweight autoregressive model in the reconstruction distribution. These techniques permit end-to-end training of models with 10+ layers of latent variables. Experiments show that our approach achieves state-of-the-art performance on standard image modelling benchmarks, can expose latent class structure in the absence of label information, and can provide convincing imputations of occluded regions in natural images.

NeurIPS Conference 2015 Conference Paper

Data Generation as Sequential Decision Making

• Philip Bachman
• Doina Precup

We connect a broad class of generative models through their shared reliance on sequential decision making. Motivated by this view, we develop extensions to an existing model, and then explore the idea further in the context of data imputation -- perhaps the simplest setting in which to investigate the relation between unconditional and conditional generative modelling. We formulate data imputation as an MDP and develop models capable of representing effective policies for it. We construct the models using neural networks and train them using a form of guided policy search. Our models generate predictions through an iterative process of feedback and refinement. We show that this approach can learn effective policies for imputation problems of varying difficulty and across multiple datasets.

EWRL Workshop 2015 Workshop Paper

Learning Policies for Data Imputation with Guided Policy Search

• Philip Bachman
• Doina Precup

We explore the relationship between directed generative models and reinforcement learning by developing a new approach to data imputation that combines ideas from both areas. We address data imputation by defining an MDP for which we construct policies parametrized by (reasonably) large neural networks. We then show how to train these policies using a form of (self) Guided Policy Search (Levine & Koltun, 2013a), which leads to maximizing a variational bound on the quality of the imputations made by our policies. Empirically, our policies perform well over a range of conditions.

ICML Conference 2015 Conference Paper

Variational Generative Stochastic Networks with Collaborative Shaping

• Philip Bachman
• Doina Precup

We develop an approach to training generative models based on unrolling a variational auto-encoder into a Markov chain, and shaping the chain’s trajectories using a technique inspired by recent work in Approximate Bayesian computation. We show that the global minimizer of the resulting objective is achieved when the generative model reproduces the target distribution. To allow finer control over the behavior of the models, we add a regularization term inspired by techniques used for regularizing certain types of policy search in reinforcement learning. We present empirical results on the MNIST and TFD datasets which show that our approach offers state-of-the-art performance, both quantitatively and from a qualitative point of view.

NeurIPS Conference 2014 Conference Paper

Learning with Pseudo-Ensembles

• Philip Bachman
• Ouais Alsharif
• Doina Precup

We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E. g. , dropout (Hinton et al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We examine the relationship of pseudo-ensembles, which involve perturbation in model-space, to standard ensemble methods and existing notions of robustness, which focus on perturbation in observation-space. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of (Socher et al, 2013) into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.

ICML Conference 2014 Conference Paper

Sample-based approximate regularization

• Philip Bachman
• Amir Massoud Farahmand
• Doina Precup

We introduce a method for regularizing linearly parameterized functions using general derivative-based penalties, which relies on sampling as well as finite-difference approximations of the relevant derivatives. We call this approach sample-based approximate regularization (SAR). We provide theoretical guarantees on the fidelity of such regularizers, compared to those they approximate, and prove that the approximations converge efficiently. We also examine the empirical performance of SAR on several datasets.

ICML Conference 2012 Conference Paper

Improved Estimation in Time Varying Models

• Doina Precup
• Philip Bachman

AAAI Conference 2011 Conference Paper

Learning Compact Representations of Time-Varying Processes

• Philip Bachman
• Doina Precup