Author name cluster

Oliver Vikbladh

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

3 papers

1 author row

RLDM Conference 2019 Conference Abstract

Episodic Memory Contributions to Model-Based Reinforcement Learning

Oliver Vikbladh
Daphna Shohamy
Nathaniel D. Daw

RL theories of biological behavior often assume that choice relies on incrementally learned running averages of previous events, either action values for model-free (MF) or one-step models for model- based (MB) accounts. A third suggestion, supported by recent findings, posits that individual trajectories are also stored as episodic memories and can later be sampled to guide choice. This raises questions for classic arguments that animals use a world model to plan actions in sequential tasks: Individual trajectories embody the same state-action-state relationships summarized in a model and might be used to similar end. To investigate the contribution of episodic memories of trial-specific outcomes during sequential choice, we scanned 30 subjects with fMRI while they performed a task that combines 2-step MDP dynamics, of the sort previously used to distinguish MB from MF, with single trial memory cues (trial unique stimuli associated with a specific reward outcomes). This allowed us to investigate whether episodic memory about the cued stimulus influences choice, how reliance of episodic cues trades off against reliance on estimates which are usually understood to be learned incrementally, and test the hypothesis whether this episodic sampling process might specifically underpin putatively incremental MB (but not MF) learning. We find behavioral evidence for an episodic choice strategy. We also show behavioral competition between this episodic strategy and the putatively incremental MB (but not MF) strategies, suggesting some shared (and other results show, item category specific) substrate. Furthermore, we demonstrate fMRI evidence that incremental MB learning shares a common hippocampal substrate with the episodic strategy, for outcome encoding. Taken together, these data provide evidence for convert episodic retrieval, rather than incremental model learning, as a possible implementation of seemingly MB choice in the human brain.

PDF Details

RLDM Conference 2017 Conference Abstract

Episodic Contributions to Model-Based Reinforcement Learning

Oliver Vikbladh
Nathaniel Daw

RL theories of human and animal behavior often assume that choice relies on incrementally learned running averages of previous events, either action values for model-free (MF) or one-step models for model-based (MB) accounts. However, a third suggestion, supported by recent findings, posits that in- dividual trajectories are also stored as separate episodic memories and can later be retrieved or sampled to guide choice. Such a third way” raises particular questions for classic arguments that animals use a cognitive map or model to plan actions in sequential tasks: Individual trajectories embody the same state-action-state relationships summarized in a world model and might be used to similar end. Conversely, their use might confound standard tests for model use. To investigate the contribution of memories for individual trials in sequential choice, we created a task that combines 2-step MDP dynamics, of the sort previously used to distinguish MB from MF, with single trial memory cues (unique objects) that also predict reward. This allowed us to investigate whether episodic information about a cued object’s previous reward influences MB or MF evaluation, and also how these effects trade off against incrementally learned estimates. 80 human subjects competed 200 trials online. In addition to significant signatures of traditional MF and MB strate- gies based on running averages, subjects displayed a significant capacity for MB planning using individually cued episodes. Furthermore, on trials that contained episodic cues (vs. those that didn’t), traditional (puta- tively incremental) MB planning was significantly reduced. This finding raises the possibility that previous interpretations of choices as reflecting running averages may instead reflect covert retrieval of individual episodes, which are replaced by explicitly cued episodes when these are provided.

PDF Details

RLDM Conference 2017 Conference Abstract

The Hippocampus as a Common Neural Substrate for Spatial Navigation and Model-Based

Oliver Vikbladh
Nathaniel Daw

The hippocampus (HC) famously supports spatial memory. However, little direct evidence cor- roborates the commonly asserted hypothesis that these spatial functions extend to map- or model-based planning, in space or otherwise. We addressed the relationship between these functions by probing both goal-directed planning and spatial memory in patients with damage to the HC following anterior temporal lobectomy (ATL). We hypothesized that if both abilities indeed share a common HC substrate, they should both be attenuated following HC damage but covary with one another the when HC is intact. 19 unilateral ATL patients and 19 controls (matched for age and IQ) participated in the study. Subjects performed a se- quential decision-making task used to differentiate reliance on model-free (MF) and model-based (MB) RL strategies. Subjects also performed a spatial navigation task that distinguished hippocampal dependent place memory”, referenced to environmental boundaries, from striatal dependent “response memory”, referenced to a discrete landmark. As predicted, patients displayed attenuated place memory but not response memory. Patients were also biased away from MB and toward MF strategies on the sequential decision-making task. Comparing the two tasks, place memory performance was correlated with the use of MB planning in the control group. Importantly, no such correlation was found in the patient group. These results show a causal role of the HC in goal-directed MB planning and speak to multiple, potentially competing, decision systems in the human brain. Finally, the demonstration that place memory correlates with MB planning only in the control group suggests that the HC may serve as a common neural substrate for flexible behavior during spatial navigation and sequential decision-making, but that following damage to this structure, differential compensatory mechanisms emerge.

PDF Details