Author name cluster

Daniel Brown

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

1 author row

NeurIPS Conference 2022 Conference Paper

Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations

Albert Wilcox
Ashwin Balakrishna
Jules Dedieu
Wyame Benslimane
Daniel Brown
Ken Goldberg

Providing densely shaped reward functions for RL algorithms is often exceedingly challenging, motivating the development of RL algorithms that can learn from easier-to-specify sparse reward functions. This sparsity poses new exploration challenges. One common way to address this problem is using demonstrations to provide initial signal about regions of the state space with high rewards. However, prior RL from demonstrations algorithms introduce significant complexity and many hyperparameters, making them hard to implement and tune. We introduce Monte Carlo Actor-Critic (MCAC), a parameter free modification to standard actor-critic algorithms which initializes the replay buffer with demonstrations and computes a modified $Q$-value by taking the maximum of the standard temporal distance (TD) target and a Monte Carlo estimate of the reward-to-go. This encourages exploration in the neighborhood of high-performing trajectories by encouraging high $Q$-values in corresponding regions of the state space. Experiments across $5$ continuous control domains suggest that MCAC can be used to significantly increase learning efficiency across $6$ commonly used RL and RL-from-demonstrations algorithms. See https: //sites. google. com/view/mcac-rl for code and supplementary material.

PDF Details

NeurIPS Conference 2020 Conference Paper

Bayesian Robust Optimization for Imitation Learning

Daniel Brown
Scott Niekum
Marek Petrik

One of the main challenges in imitation learning is determining what action an agent should take when outside the state distribution of the demonstrations. Inverse reinforcement learning (IRL) can enable generalization to new states by learning a parameterized reward function, but these approaches still face uncertainty over the true reward function and corresponding optimal policy. Existing safe imitation learning approaches based on IRL deal with this uncertainty using a maxmin framework that optimizes a policy under the assumption of an adversarial reward function, whereas risk-neutral IRL approaches either optimize a policy for the mean or MAP reward function. While completely ignoring risk can lead to overly aggressive and unsafe policies, optimizing in a fully adversarial sense is also problematic as it can lead to overly conservative policies that perform poorly in practice. To provide a bridge between these two extremes, we propose Bayesian Robust Optimization for Imitation Learning (BROIL). BROIL leverages Bayesian reward function inference and a user specific risk tolerance to efficiently optimize a robust policy that balances expected return and conditional value at risk. Our empirical results show that BROIL provides a natural way to interpolate between return-maximizing and risk-minimizing behaviors and outperforms existing risk-sensitive and risk-neutral inverse reinforcement learning algorithms.

PDF Details

AAAI Conference 2018 Conference Paper

Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning

Daniel Brown
Scott Niekum

In the ﬁeld of reinforcement learning there has been recent progress towards safety and high-conﬁdence bounds on policy performance. However, to our knowledge, no practical methods exist for determining high-conﬁdence policy performance bounds in the inverse reinforcement learning setting— where the true reward function is unknown and only samples of expert behavior are given. We propose a sampling method based on Bayesian inverse reinforcement learning that uses demonstrations to determine practical high-conﬁdence upper bounds on the α-worst-case difference in expected return between any evaluation policy and the optimal policy under the expert’s unknown reward function. We evaluate our proposed bound on both a standard grid navigation task and a simulated driving task and achieve tighter and more accurate bounds than a feature count-based baseline. We also give examples of how our proposed bound can be utilized to perform riskaware policy selection and risk-aware policy improvement. Because our proposed bound requires several orders of magnitude fewer demonstrations than existing high-conﬁdence bounds, it is the ﬁrst practical method that allows agents that learn from demonstration to express conﬁdence in the quality of their learned policy.

PDF Details

AIIM Journal 2018 Journal Article

Temporal case-based reasoning for type 1 diabetes mellitus bolus insulin decision support

Daniel Brown
Arantza Aldea
Rachel Harrison
Clare Martin
Ian Bayley

Individuals with type 1 diabetes have to monitor their blood glucose levels, determine the quantity of insulin required to achieve optimal glycaemic control and administer it themselves subcutaneously, multiple times per day. To help with this process bolus calculators have been developed that suggest the appropriate dose. However these calculators do not automatically adapt to the specific circumstances of an individual and require fine-tuning of parameters, a process that often requires the input of an expert. To overcome the limitations of the traditional methods this paper proposes the use of an artificial intelligence technique, case-based reasoning, to personalise the bolus calculation. A novel aspect of our approach is the use of temporal sequences to take into account preceding events when recommending the bolus insulin doses rather than looking at events in isolation. The in silico results described in this paper show that given the initial conditions of the patient, the temporal retrieval algorithm identifies the most suitable case for reuse. Additionally through insulin-on-board adaptation and postprandial revision, the approach is able to learn and improve bolus predictions, reducing the blood glucose risk index by up to 27% after three revisions of a bolus solution.

Details DOI