Arrow Research search

Author name cluster

Katja Hofmann

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

34 papers
2 author rows

Possible papers

34

ICML Conference 2025 Conference Paper

Scaling Laws for Pre-training Agents and World Models

  • Tim Pearce
  • Tabish Rashid
  • David Bignell
  • Raluca Georgescu
  • Sam Devlin
  • Katja Hofmann

The performance of embodied agents has been shown to improve by increasing model parameters, dataset size, and compute. This has been demonstrated in domains from robotics to video games, when generative learning objectives on offline datasets (pre-training) are used to model an agent’s behavior (imitation learning) or their environment (world modeling). This paper characterizes the role of scale in these tasks more precisely. Going beyond the simple intuition that ‘bigger is better’, we show that the same types of power laws found in language modeling also arise in world modeling and imitation learning (e. g. between loss and optimal model size). However, the coefficients of these laws are heavily influenced by the tokenizer, task & architecture – this has important implications on the optimal sizing of models and data.

Details

ICLR Conference 2023 Conference Paper

Imitating Human Behaviour with Diffusion Models

  • Tim Pearce
  • Tabish Rashid
  • Anssi Kanervisto
  • David Bignell
  • Mingfei Sun 0001
  • Raluca Georgescu
  • Sergio Valcarcel Macua
  • Shan Zheng Tan

Diffusion models have emerged as powerful generative models in the text-to-image domain. This paper studies their application as observation-to-action models for imitating human behaviour in sequential environments. Human behaviour is stochastic and multimodal, with structured correlations between action dimensions. Meanwhile, standard modelling choices in behaviour cloning are limited in their expressiveness and may introduce bias into the cloned policy. We begin by pointing out the limitations of these choices. We then propose that diffusion models are an excellent fit for imitating human behaviour, since they learn an expressive distribution over the joint action space. We introduce several innovations to make diffusion models suitable for sequential environments; designing suitable architectures, investigating the role of guidance, and developing reliable sampling strategies. Experimentally, diffusion models closely match human demonstrations in a simulated robotic control task and a modern 3D gaming environment.

Details

AAMAS Conference 2023 Conference Paper

Trust Region Bounds for Decentralized PPO Under Non-stationarity

  • Mingfei Sun
  • Sam Devlin
  • Jacob Beck
  • Katja Hofmann
  • Shimon Whiteson

We present trust region bounds for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary. This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, which both rely on independent ratios, i. e. , computing probability ratios separately for each agent’s policy. We show that, despite the nonstationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over all decentralized policies. We also show this trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training, providing a theoretical foundation for proximal ratio clipping. Finally, our empirical results support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training, and tuning the hyperparameters with regards to the number of agents, as predicted by our theoretical analysis.

PDF

NeurIPS Conference 2022 Conference Paper

Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification

  • Massimiliano Patacchiola
  • John Bronskill
  • Aliaksandra Shysheya
  • Katja Hofmann
  • Sebastian Nowozin
  • Richard Turner

Recent years have seen a growth in user-centric applications that require effective knowledge transfer across tasks in the low-data regime. An example is personalization, where a pretrained system is adapted by learning on small amounts of labeled data belonging to a specific user. This setting requires high accuracy under low computational complexity, therefore the Pareto frontier of accuracy vs. adaptation cost plays a crucial role. In this paper we push this Pareto frontier in the few-shot image classification setting with a key contribution: a new adaptive block called Contextual Squeeze-and-Excitation (CaSE) that adjusts a pretrained neural network on a new task to significantly improve performance with a single forward pass of the user data (context). We use meta-trained CaSE blocks to conditionally adapt the body of a network and a fine-tuning routine to adapt a linear head, defining a method called UpperCaSE. UpperCaSE achieves a new state-of-the-art accuracy relative to meta-learners on the 26 datasets of VTAB+MD and on a challenging real-world personalization benchmark (ORBIT), narrowing the gap with leading fine-tuning methods with the benefit of orders of magnitude lower adaptation cost.

PDF Details

AAAI Conference 2022 Conference Paper

Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency

  • Mingfei Sun
  • Sam Devlin
  • Katja Hofmann
  • Shimon Whiteson

Sample efficiency is crucial for imitation learning methods to be applicable in real-world applications. Many studies improve sample efficiency by extending adversarial imitation to be off-policy regardless of the fact that these off-policy extensions could either change the original objective or involve complicated optimization. We revisit the foundation of adversarial imitation and propose an off-policy sample efficient approach that requires no adversarial training or min-max optimization. Our formulation capitalizes on two key insights: (1) the similarity between the Bellman equation and the stationary state-action distribution equation allows us to derive a novel temporal difference (TD) learning approach; and (2) the use of a deterministic policy simplifies the TD learning. Combined, these insights yield a practical algorithm, Deterministic and Discriminative Imitation (D2-Imitation), which operates by first partitioning samples into two replay buffers and then learning a deterministic policy via off-policy reinforcement learning. Our empirical results show that D2-Imitation is effective in achieving good sample efficiency, outperforming several off-policy extension approaches of adversarial imitation on many control tasks.

PDF Details

ICML Conference 2022 Conference Paper

Interactively Learning Preference Constraints in Linear Bandits

  • David Lindner
  • Sebastian Tschiatschek
  • Katja Hofmann
  • Andreas Krause 0001

We study sequential decision-making with known rewards and unknown constraints, motivated by situations where the constraints represent expensive-to-evaluate human preferences, such as safe and comfortable driving behavior. We formalize the challenge of interactively learning about these constraints as a novel linear bandit problem which we call constrained linear best-arm identification. To solve this problem, we propose the Adaptive Constraint Learning (ACOL) algorithm. We provide an instance-dependent lower bound for constrained linear best-arm identification and show that ACOL’s sample complexity matches the lower bound in the worst-case. In the average case, ACOL’s sample complexity bound is still significantly tighter than bounds of simpler approaches. In synthetic experiments, ACOL performs on par with an oracle solution and outperforms a range of baselines. As an application, we consider learning constraints to represent human preferences in a driving simulation. ACOL is significantly more sample efficient than alternatives for this application. Further, we find that learning preferences as constraints is more robust to changes in the driving scenario than encoding the preferences directly in the reward function.

Details

NeurIPS Conference 2022 Conference Paper

Uni[MASK]: Unified Inference in Sequential Decision Problems

  • Micah Carroll
  • Orr Paradise
  • Jessy Lin
  • Raluca Georgescu
  • Mingfei Sun
  • David Bignell
  • Stephanie Milani
  • Katja Hofmann

Randomly masking and predicting word tokens has been a successful approach in pre-training language models for a variety of downstream tasks. In this work, we observe that the same idea also applies naturally to sequential decision making, where many well-studied tasks like behavior cloning, offline RL, inverse dynamics, and waypoint conditioning correspond to different sequence maskings over a sequence of states, actions, and returns. We introduce the UniMASK framework, which provides a unified way to specify models which can be trained on many different sequential decision making tasks. We show that a single UniMASK model is often capable of carrying out many tasks with performance similar to or better than single-task models. Additionally, after fine-tuning, our UniMASK models consistently outperform comparable single-task models.

PDF Details

AAMAS Conference 2021 Conference Paper

Deep Interactive Bayesian Reinforcement Learning via Meta-Learning

  • Luisa Zintgraf
  • Sam Devlin
  • Kamil Ciosek
  • Shimon Whiteson
  • Katja Hofmann

Agents that interact with other agents often do not know a priori what the other agents’ strategies are, but have to maximise their own online return while interacting with and learning about others. The optimal adaptive behaviour under uncertainty over the other agents’ strategies w. r. t. some prior can in principle be computed using the Interactive Bayesian Reinforcement Learning framework. Unfortunately, doing so is intractable in most settings, and existing approximation methods are restricted to small tasks. To overcome this, we propose to meta-learn (alongside the policy) approximate belief inference by combining sequential and hierarchical VAEs. We show empirically that our approach can learn a factorised belief model that separates the other agent’s permanent and temporal structure, and outperforms methods that sample from the approximate posterior or do not have this hierarchical structure. A full version of this work can be found in Zintgraf et al. [30].

PDF

AAMAS Conference 2021 Conference Paper

Evaluating the Robustness of Collaborative Agents

  • Paul Knott
  • Micah Carroll
  • Sam Devlin
  • Kamil Ciosek
  • Katja Hofmann
  • Anca Dragan
  • Rohin Shah

Artificial agents trained by deep reinforcement learning will likely encounter novel situations after deployment that were never seen during training. Our agent must be robust to handle such situations well. However, if we cannot rely on the average training or validation reward as a metric, then how can we effectively evaluate robustness? We take inspiration from the practice of unit testing in software engineering. Specifically, we suggest that when designing AI agents that collaborate with humans, designers should search for potential edge cases in possible partner behavior and possible states encountered, and write tests which check that the behavior of the agent in these edge cases is reasonable. We apply this methodology to build a suite of unit tests for the Overcooked-AI environment, and use this test suite to evaluate three proposals for improving robustness. We find that the test suite provides significant insight into the effects of these proposals that were generally not revealed by looking solely at the average validation reward. For our full paper, see arxiv. org/abs/2101. 05507.

PDF

ICML Conference 2021 Conference Paper

Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning

  • Luisa M. Zintgraf
  • Leo Feng
  • Cong Lu
  • Maximilian Igl
  • Kristian Hartikainen
  • Katja Hofmann
  • Shimon Whiteson

To rapidly learn a new task, it is often essential for agents to explore efficiently - especially when performance matters from the first timestep. One way to learn such behaviour is via meta-learning. Many existing methods however rely on dense rewards for meta-training, and can fail catastrophically if the rewards are sparse. Without a suitable reward signal, the need for exploration during meta-training is exacerbated. To address this, we propose HyperX, which uses novel reward bonuses for meta-training to explore in approximate hyper-state space (where hyper-states represent the environment state and the agent’s task belief). We show empirically that HyperX meta-learns better task-exploration and adapts more successfully to new tasks than existing methods.

Details

NeurIPS Conference 2021 Conference Paper

Grounding Spatio-Temporal Language with Transformers

  • Tristan Karch
  • Laetitia Teodorescu
  • Katja Hofmann
  • Clément Moulin-Frier
  • Pierre-Yves Oudeyer

Language is an interface to the outside world. In order for embodied agents to use it, language must be grounded in other, sensorimotor modalities. While there is an extended literature studying how machines can learn grounded language, the topic of how to learn spatio-temporal linguistic concepts is still largely uncharted. To make progress in this direction, we here introduce a novel spatio-temporal language grounding task where the goal is to learn the meaning of spatio-temporal descriptions of behavioral traces of an embodied agent. This is achieved by training a truth function that predicts if a description matches a given history of observations. The descriptions involve time-extended predicates in past and present tense as well as spatio-temporal references to objects in the scene. To study the role of architectural biases in this task, we train several models including multimodal Transformer architectures; the latter implement different attention computations between words and objects across space and time. We test models on two classes of generalization: 1) generalization to new sentences, 2) generalization to grammar primitives. We observe that maintaining object identity in the attention computation of our Transformers is instrumental to achieving good performance on generalization overall, and that summarizing object traces in a single token has little influence on performance. We then discuss how this opens new perspectives for language-guided autonomous embodied agents.

PDF Details

NeurIPS Conference 2021 Conference Paper

Memory Efficient Meta-Learning with Large Images

  • John Bronskill
  • Daniela Massiceti
  • Massimiliano Patacchiola
  • Katja Hofmann
  • Sebastian Nowozin
  • Richard Turner

Meta learning approaches to few-shot classification are computationally efficient at test time, requiring just a few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task's entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, or trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for a task can be decomposed into a sum of gradients over the task's training images. This enables us to perform a forward pass on a task's entire training set but realize significant memory savings by back-propagating only a random subset of these images which we show is an unbiased approximation of the full gradient. We use LITE to train meta-learners and demonstrate new state-of-the-art accuracy on the real-world ORBIT benchmark and 3 of the 4 parts of the challenging VTAB+MD benchmark relative to leading meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at a fraction of the test-time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.

PDF Details

ICML Conference 2021 Conference Paper

Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation

  • Sam Devlin
  • Raluca Georgescu
  • Ida Momennejad
  • Jaroslaw Rzepecki
  • Evelyn Zuniga
  • Gavin Costello
  • Guy Leroy
  • Ali Shaw

A key challenge on the path to developing agents that learn complex human-like behavior is the need to quickly and accurately quantify human-likeness. While human assessments of such behavior can be highly accurate, speed and scalability are limited. We address these limitations through a novel automated Navigation Turing Test (ANTT) that learns to predict human judgments of human-likeness. We demonstrate the effectiveness of our automated NTT on a navigation task in a complex 3D environment. We investigate six classification models to shed light on the types of architectures best suited to this task, and validate them against data collected through a human NTT. Our best models achieve high accuracy when distinguishing true human and agent behavior. At the same time, we show that predicting finer-grained human assessment of agents’ progress towards human-like behavior remains unsolved. Our work takes an important step towards agents that more effectively learn complex human-like behavior.

Details

UAI Conference 2021 Conference Paper

Strategically efficient exploration in competitive multi-agent reinforcement learning

  • Robert Tyler Loftin
  • Aadirupa Saha
  • Sam Devlin
  • Katja Hofmann

High sample complexity remains a barrier to the application of reinforcement learning (RL), particularly in multi-agent systems. A large body of work has demonstrated that exploration mechanisms based on the principle of optimism under uncertainty can significantly improve the sample efficiency of RL in single agent tasks. This work seeks to understand the role of optimistic exploration in non-cooperative multi-agent settings. We will show that, in zero-sum games, optimistic exploration can cause the learner to waste time sampling parts of the state space that are irrelevant to strategic play, as they can only be reached through cooperation between both players. To address this issue, we introduce a formal notion of strategically efficient exploration in Markov games, and use this to develop two strategically efficient learning algorithms for finite Markov games. We demonstrate that these methods can be significantly more sample efficient than their optimistic counterparts.

Details

ICML Conference 2021 Conference Paper

TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL

  • Clément Romac
  • Rémy Portelas
  • Katja Hofmann
  • Pierre-Yves Oudeyer

Training autonomous agents able to generalize to multiple tasks is a key target of Deep Reinforcement Learning (DRL) research. In parallel to improving DRL algorithms themselves, Automatic Curriculum Learning (ACL) study how teacher algorithms can train DRL agents more efficiently by adapting task selection to their evolving abilities. While multiple standard benchmarks exist to compare DRL agents, there is currently no such thing for ACL algorithms. Thus, comparing existing approaches is difficult, as too many experimental parameters differ from paper to paper. In this work, we identify several key challenges faced by ACL algorithms. Based on these, we present TeachMyAgent (TA), a benchmark of current ACL algorithms leveraging procedural task generation. It includes 1) challenge-specific unit-tests using variants of a procedural Box2D bipedal walker environment, and 2) a new procedural Parkour environment combining most ACL challenges, making it ideal for global performance assessment. We then use TeachMyAgent to conduct a comparative study of representative existing approaches, showcasing the competitiveness of some ACL algorithms that do not use expert knowledge. We also show that the Parkour environment remains an open problem. We open-source our environments, all studied ACL algorithms (collected from open-source code or re-implemented), and DRL students in a Python package available at https: //github. com/flowersteam/TeachMyAgent.

Details

JMLR Journal 2021 Journal Article

VariBAD: Variational Bayes-Adaptive Deep RL via Meta-Learning

  • Luisa Zintgraf
  • Sebastian Schulze
  • Cong Lu
  • Leo Feng
  • Maximilian Igl
  • Kyriacos Shiarlis
  • Yarin Gal
  • Katja Hofmann

Trading off exploration and exploitation in an unknown environment is key to maximising expected online return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but also on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn approximately Bayes-optimal policies for complex tasks. VariBAD simultaneously meta-learns a variational auto-encoder to perform approximate inference, and a policy that incorporates task uncertainty directly during action selection by conditioning on both the environment state and the approximate belief. In two toy domains, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo tasks widely used in meta-RL and show that it achieves higher online return than existing methods. On the recently proposed Meta-World ML1 benchmark, variBAD achieves state of the art results by a large margin, fully solving two out of the three ML1 tasks for the first time. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2021. ( edit, beta )

PDF Details

ICLR Conference 2020 Conference Paper

AMRL: Aggregated Memory For Reinforcement Learning

  • Jacob Beck
  • Kamil Ciosek
  • Sam Devlin
  • Sebastian Tschiatschek
  • Cheng Zhang 0005
  • Katja Hofmann

In many partially observable scenarios, Reinforcement Learning (RL) agents must rely on long-term memory in order to learn an optimal policy. We demonstrate that using techniques from NLP and supervised learning fails at RL tasks due to stochasticity from the environment and from exploration. Utilizing our insights on the limitations of traditional memory methods in RL, we propose AMRL, a class of models that can learn better policies with greater sample efficiency and are resilient to noisy inputs. Specifically, our models use a standard memory module to summarize short-term context, and then aggregate all prior states from the standard model without respect to order. We show that this provides advantages both in terms of gradient decay and signal-to-noise ratio over time. Evaluating in Minecraft and maze environments that test long-term memory, we find that our model improves average return by 19% over a baseline that has the same number of parameters and by 9% over a stronger baseline that has far more parameters.

Details

IJCAI Conference 2020 Conference Paper

Automatic Curriculum Learning For Deep RL: A Short Survey

  • Rémy Portelas
  • Cédric Colas
  • Lilian Weng
  • Katja Hofmann
  • Pierre-Yves Oudeyer

Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL). These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. To do so, ACL mechanisms can act on many aspects of learning problems. They can optimize domain randomization for Sim2Real transfer, organize task presentations in multi-task robotic settings, order sequences of opponents in multi-agent scenarios, etc. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.

PDF Details DOI

ICLR Conference 2020 Conference Paper

Conservative Uncertainty Estimation By Fitting Prior Networks

  • Kamil Ciosek
  • Vincent Fortuin
  • Ryota Tomioka
  • Katja Hofmann
  • Richard E. Turner

Obtaining high-quality uncertainty estimates is essential for many applications of deep neural networks. In this paper, we theoretically justify a scheme for estimating uncertainties, based on sampling from a prior distribution. Crucially, the uncertainty estimates are shown to be conservative in the sense that they never underestimate a posterior uncertainty obtained by a hypothetical Bayesian algorithm. We also show concentration, implying that the uncertainty estimates converge to zero as we get more data. Uncertainty estimates obtained from random priors can be adapted to any deep network architecture and trained using standard supervised learning pipelines. We provide experimental evaluation of random priors on calibration and out-of-distribution detection on typical computer vision tasks, demonstrating that they outperform deep ensembles in practice.

Details

ICLR Conference 2020 Conference Paper

VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

  • Luisa M. Zintgraf
  • Kyriacos Shiarlis
  • Maximilian Igl
  • Sebastian Schulze
  • Yarin Gal
  • Katja Hofmann
  • Shimon Whiteson

Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent’s uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncer- tainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher online return than existing methods.

Details

NeurIPS Conference 2019 Conference Paper

Better Exploration with Optimistic Actor Critic

  • Kamil Ciosek
  • Quan Vuong
  • Robert Loftin
  • Katja Hofmann

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address this problem both theoretically and empirically. On the theoretical side, we identify two phenomena preventing efficient exploration in existing state-of-the-art algorithms such as Soft Actor Critic. First, combining a greedy actor update with a pessimistic estimate of the critic leads to the avoidance of actions that the agent does not know about, a phenomenon we call pessimistic underexploration. Second, current algorithms are directionally uninformed, sampling actions with equal probability in opposite directions from the current mean. This is wasteful, since we typically need actions taken along certain directions much more than others. To address both of these phenomena, we introduce a new algorithm, Optimistic Actor Critic, which approximates a lower and upper confidence bound on the state-action value function. This allows us to apply the principle of optimism in the face of uncertainty to perform directed exploration using the upper bound while still using the lower bound to avoid overestimation. We evaluate OAC in several challenging continuous control tasks, achieving state-of the art sample efficiency.

PDF Details

ICML Conference 2019 Conference Paper

Fast Context Adaptation via Meta-Learning

  • Luisa M. Zintgraf
  • Kyriacos Shiarlis
  • Vitaly Kurin
  • Katja Hofmann
  • Shimon Whiteson

We propose CAVIA for meta-learning, a simple extension to MAML that is less prone to meta-overfitting, easier to parallelise, and more interpretable. CAVIA partitions the model parameters into two parts: context parameters that serve as additional input to the model and are adapted on individual tasks, and shared parameters that are meta-trained and shared across tasks. At test time, only the context parameters are updated, leading to a low-dimensional task representation. We show empirically that CAVIA outperforms MAML for regression, classification, and reinforcement learning. Our experiments also highlight weaknesses in current benchmarks, in that the amount of adaptation needed in some cases is small.

Details

NeurIPS Conference 2019 Conference Paper

Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck

  • Maximilian Igl
  • Kamil Ciosek
  • Yingzhen Li
  • Sebastian Tschiatschek
  • Cheng Zhang
  • Sam Devlin
  • Katja Hofmann

The ability for policies to generalize to new environments is key to the broad application of RL agents. A promising approach to prevent an agent’s policy from overfitting to a limited set of training environments is to apply regularization techniques originally developed for supervised learning. However, there are stark differences between supervised learning and RL. We discuss those differences and propose modifications to existing regularization techniques in order to better adapt them to RL. In particular, we focus on regularization techniques relying on the injection of noise into the learned function, a family that includes some of the most widely used approaches such as Dropout and Batch Normalization. To adapt them to RL, we propose Selective Noise Injection (SNI), which maintains the regularizing effect the injected noise has, while mitigating the adverse effects it has on the gradient quality. Furthermore, we demonstrate that the Information Bottleneck (IB) is a particularly well suited regularization technique for RL as it is effective in the low-data regime encountered early on in training RL agents. Combining the IB with SNI, we significantly outperform current state of the art results, including on the recently proposed generalization benchmark Coinrun.

PDF Details

RLDM Conference 2019 Conference Abstract

Multi-task Reinforcement Learning and the MineRL Competition

  • Katja Hofmann

Multi-task reinforcement learning (RL) aims to develop approaches that learn to perform well across a range of related tasks, instead of specializing to a single task. This has high potential for real-world appli- cations, where sharing data across tasks can dramatically improve data efficiency to make RL approaches economically viable. In this talk, I present two novel approaches that leverage learned task embeddings to exploit multi-task structure for sample-efficient learning. Last, but not least, I will outline directions towards sample-efficient RL, and introduce the MineRL competition which is designed to foster research in this ex- citing and timely research area.

PDF Details

NeurIPS Conference 2019 Conference Paper

Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning

  • David Janz
  • Jiri Hron
  • Przemysław Mazur
  • Katja Hofmann
  • José Miguel Hernández-Lobato
  • Sebastian Tschiatschek

Posterior sampling for reinforcement learning (PSRL) is an effective method for balancing exploration and exploitation in reinforcement learning. Randomised value functions (RVF) can be viewed as a promising approach to scaling PSRL. However, we show that most contemporary algorithms combining RVF with neural network function approximation do not possess the properties which make PSRL effective, and provably fail in sparse reward problems. Moreover, we find that propagation of uncertainty, a property of PSRL previously thought important for exploration, does not preclude this failure. We use these insights to design Successor Uncertainties (SU), a cheap and easy to implement RVF algorithm that retains key properties of PSRL. SU is highly effective on hard tabular exploration benchmarks. Furthermore, on the Atari 2600 domain, it surpasses human performance on 38 of 49 games tested (achieving a median human normalised score of 2. 09), and outperforms its closest RVF competitor, Bootstrapped DQN, on 36 of those.

PDF Details

IJCAI Conference 2018 Conference Paper

Advancements in Dueling Bandits

  • Yanan Sui
  • Masrour Zoghi
  • Katja Hofmann
  • Yisong Yue

The dueling bandits problem is an online learning framework where learning happens ``on-the-fly'' through preference feedback, i. e. , from comparisons between a pair of actions. Unlike conventional online learning settings that require absolute feedback for each action, the dueling bandits framework assumes only the presence of (noisy) binary feedback about the relative quality of each pair of actions. The dueling bandits problem is well-suited for modeling settings that elicit subjective or implicit human feedback, which is typically more reliable in preference form. In this survey, we review recent results in the theories, algorithms, and applications of the dueling bandits problem. As an emerging domain, the theories and algorithms of dueling bandits have been intensively studied during the past few years. We provide an overview of recent advancements, including algorithmic advances and applications. We discuss extensions to standard problem formulation and novel application areas, highlighting key open research questions in our discussion.

PDF Details

EWRL Workshop 2018 Workshop Paper

Combining No-regret and Q-learning

  • Ian Kash
  • Katja Hofmann

Most reinforcement learning algorithms do not provide guarantees in settings with multiple agents or partial observability. A notable exception is Counterfactual Regret Minimization (CFR), which provides both strong convergence guarantees and empirical results in settings like poker. We seek to understand how these guarantees could be achieved more broadly. To take a first step in this direction, we introduce a simple algorithm, local no-regret learning (LONR), which captures the spirit of CFR, but can be applied in settings without a terminal state. We prove its convergence for the basic case of MDPs and discuss research directions to extend our results to address richer settings with multiple agents, partial observability, and sampling.

PDF Details

EWRL Workshop 2018 Workshop Paper

Learning good policies from suboptimal demonstrations

  • Yuxiang Li
  • Katja Hofmann
  • Ian Kash

Imitating an expert policy is one way to boost reinforcement learning algorithms which, in most cases, rely on random exploration and a huge amount of data. While some major drawbacks of imitation learning, such as compound errors, have been addressed, most results make an implicit hypothesis of having a good (or desired) expert which is usually hard to get in practice. To overcome this obstacle, this paper focuses on learning good policies from suboptimal demonstration data. We systematically investigate the performance of a recent approach under varying assumptions on demonstration quality and show that it performs poorly with suboptimal demonstration data. We then demonstrate the potential to overcome this issue through a performance comparison between learner and demonstrator.

PDF Details

UAI Conference 2018 Conference Paper

Meta Reinforcement Learning with Latent Variable Gaussian Processes

  • Steindór Sæmundsson
  • Katja Hofmann
  • Marc Peter Deisenroth

Learning from small data sets is critical in many practical applications where data collection is time consuming or expensive, e. g. , robotics, animal experiments or drug design. Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Often, this relationship between tasks is hard coded or relies in some other way on human expertise. In this paper, we frame meta learning as a hierarchical latent variable model and infer the relationship between tasks automatically from data. We apply our framework in a modelbased reinforcement learning setting and show that our meta-learning model effectively generalizes to novel tasks by identifying how new tasks relate to prior ones from minimal data. This results in up to a 60% reduction in the average interaction time needed to solve tasks compared to strong baselines.

Details

AAMAS Conference 2017 Conference Paper

Asynchronous Data Aggregation for Training End to End Visual Control Networks

  • Mathew Monfort
  • Matthew Johnson
  • Aude Oliva
  • Katja Hofmann

Robust training of deep neural networks requires a large amount of data. However gathering and labeling this data can be expensive and determining which distribution of features are needed for training is not a trivial problem. This is compounded when training neural networks for autonomous navigation in continuous nondeterministic environments using only visual input. Increasing the quantity of demonstrated data does not solve this problem as the demonstrated sequences of actions are not guaranteed to produce the same outcomes and slight changes in orientation generate drastically different visual representations. This results in a training set with a different distribution than what the agent will typically encounter in application. Here, we develop a method that can grow a training set from the same distribution as the agent’s experiences and capture useful features not found in demonstrated behavior. Additionally, we show that our approach scales to efficiently handle complex tasks that require a large amount of data (experiences) for training. Concretely, we propose the deep asynchronous Dagger framework, which combines the Dagger algorithm with an asynchronous actor-learner architecture for parallel dataset aggregation and network policy learning. We apply our method to the task of navigating 3D mazes in Minecraft with randomly changing block types and analyze our results.

PDF

AAAI Conference 2016 Conference Paper

Collective Noise Contrastive Estimation for Policy Transfer Learning

  • Weinan Zhang
  • Ulrich Paquet
  • Katja Hofmann

We address the problem of learning behaviour policies to optimise online metrics from heterogeneous usage data. While online metrics, e. g. , click-through rate, can be optimised effectively using exploration data, such data is costly to collect in practice, as it temporarily degrades the user experience. Leveraging related data sources to improve online performance would be extremely valuable, but is not possible using current approaches. We formulate this task as a policy transfer learning problem, and propose a first solution, called collective noise contrastive estimation (collective NCE). NCE is an efficient solution to approximating the gradient of a logsoftmax objective. Our approach jointly optimises embeddings of heterogeneous data to transfer knowledge from the source domain to the target domain. We demonstrate the effectiveness of our approach by learning an effective policy for an online radio station jointly from user-generated playlists, and usage data collected in an exploration bucket.

PDF Details

EWRL Workshop 2016 Workshop Paper

Decoding multitask DQN in the world of Minecraft

  • Lydia Liu
  • Urun Dogan
  • Katja Hofmann

Multitask networks that can play multiple Atari games at expert level have been successfully trained using supervised learning from several single task Deep Q Networks (DQN). However, such networks are not be able to exploit the high level similarity between games or learn common representations of game states. In fact, learned representations were shown to be separable given the game. In our work, we show that with sufficient similarity between tasks, we can train a multitask extension of DQN (MDQN) which shares representations across tasks without loss of performance. To this end, we construct a novel set of tasks with shared characteristics in Minecraft, a complex 3D world, and are able to demonstrate meaningful representation sharing between different related tasks. Sharing representations for similar tasks has paramount importance for transfer learning and lifelong learning. We envision results of this work as a stepping stone to novel lifelong learning approaches.

PDF Details

EWRL Workshop 2016 Workshop Paper

Memory Lens: How Much Memory Does an Agent Use?

  • Christoph Dann
  • Katja Hofmann
  • Sebastian Nowozin

We propose a new method to study the internal memory used by reinforcement learning policies. We estimate the amount of relevant past information by estimating mutual information between behavior histories and the current action of an agent. We perform this estimation in the passive setting, that is, we do not intervene but merely observe the natural behavior of the agent. Moreover, we provide a theoretical justi cation for our approach by showing that it yields an implementation-independent lower bound on the minimal memory capacity of any agent that implement the observed policy. We demonstrate our approach by estimating the use of memory of DQN policies on concatenated Atari frames, demonstrating sharply di erent use of memory across 49 games. The study of memory as information that ows from the past to the current action opens avenues to understand and improve successful reinforcement learning algorithms.

PDF Details

IJCAI Conference 2016 Conference Paper

The Malmo Platform for Artificial Intelligence Experimentation

  • Matthew Johnson
  • Katja Hofmann
  • Tim Hutton
  • David Bignell

We present Project Malmo - an AI experimentation platform built on top of the popular computer game Minecraft, and designed to support fundamental research in artificial intelligence. As the AI research community pushes for artificial general intelligence (AGI), experimentation platforms are needed that support the development of flexible agents that learn to solve diverse tasks in complex environments. Minecraft is an ideal foundation for such a platform, as it exposes agents to complex 3D worlds, coupled with infinitely varied game-play. Project Malmo provides a sophisticated abstraction layer on top of Minecraft that supports a wide range of experimentation scenarios, ranging from navigation and survival to collaboration and problem solving tasks. In this demo we present the Malmo platform and its capabilities. The platform is publicly released as open source software at IJCAI, to support openness and collaboration in AI research.

PDF Details
v2026.09.13