Samuel Gershman

ICLR Conference 2025 Conference Paper

Do Mice Grok? Glimpses of Hidden Progress in Sensory Cortex

• Tanishq Kumar
• Blake Bordelon
• Cengiz Pehlevan
• Venkatesh N. Murthy
• Samuel Gershman

Does learning of task-relevant representations stop when behavior stops changing? Motivated by recent work in machine learning and the intuitive observation that human experts continue to learn after mastery, we hypothesize that task-specific representation learning in cortex can continue, even when behavior saturates. In a novel reanalysis of recently published neural data, we find evidence for such learning in posterior piriform cortex of mice following continued training on a task, long after behavior saturates at near-ceiling performance ("overtraining"). We demonstrate that class representations in cortex continue to separate during overtraining, so that examples that were incorrectly classified at the beginning of overtraining can abruptly be correctly classified later on, despite no changes in behavior during that time. We hypothesize this hidden learning takes the form of approximate margin maximization; we validate this and other predictions in the neural data, as well as build and interpret a simple synthetic model that recapitulates these phenomena. We conclude by demonstrating how this model of late-time feature learning implies an explanation for the empirical puzzle of overtraining reversal in animal learning, where task-specific representations are more robust to particular task changes because the learned features can be reused.

ICLR Conference 2024 Conference Paper

Grokking as the transition from lazy to rich training dynamics

• Tanishq Kumar
• Blake Bordelon
• Samuel Gershman
• Cengiz Pehlevan

We propose that the grokking phenomenon, where the train loss of a neural network decreases much earlier than its test loss, can arise due to a neural network transitioning from lazy training dynamics to a rich, feature learning regime. To illustrate this mechanism, we study the simple setting of vanilla gradient descent on a polynomial regression problem with a two layer neural network which exhibits grokking without regularization in a way that cannot be explained by existing theories. We identify sufficient statistics for the test loss of such a network, and tracking these over training reveals that grokking arises in this setting when the network first attempts to fit a kernel regression solution with its initial features, followed by late-time feature learning where a generalizing solution is identified after train loss is already low. We find that the key determinants of grokking are the rate of feature learning---which can be controlled precisely by parameters that scale the network output---and the alignment of the initial features with the target function $y(x)$. We argue this delayed generalization arises when (1) the top eigenvectors of the initial neural tangent kernel and the task labels $y(x)$ are misaligned, but (2) the dataset size is large enough so that it is possible for the network to generalize eventually, but not so large that train loss perfectly tracks test loss at all epochs, and (3) the network begins training in the lazy regime so does not learn features immediately. We conclude with evidence that this transition from lazy (linear model) to rich training (feature learning) can control grokking in more general settings, like on MNIST, one-layer Transformers, and student-teacher networks.

ICLR Conference 2022 Conference Paper

Hybrid Memoised Wake-Sleep: Approximate Inference at the Discrete-Continuous Interface

• Tuan Anh Le 0001
• Katherine M. Collins
• Luke Hewitt
• Kevin Ellis
• N. Siddharth 0001
• Samuel Gershman
• Joshua B. Tenenbaum

Modeling complex phenomena typically involves the use of both discrete and continuous variables. Such a setting applies across a wide range of problems, from identifying trends in time-series data to performing effective compositional scene understanding in images. Here, we propose Hybrid Memoised Wake-Sleep (HMWS), an algorithm for effective inference in such hybrid discrete-continuous models. Prior approaches to learning suffer as they need to perform repeated expensive inner-loop discrete inference. We build on a recent approach, Memoised Wake-Sleep (MWS), which alleviates part of the problem by memoising discrete variables, and extend it to allow for a principled and effective way to handle continuous variables by learning a separate recognition model used for importance-sampling based approximate inference and marginalization. We evaluate HMWS in the GP-kernel learning and 3D scene understanding domains, and show that it outperforms current state-of-the-art inference methods.

RLDM Conference 2019 Conference Abstract

Developmental change in the use of causal inference to guide reinforcement learning

• Kate Nussenbaum
• Alexandra Cohen
• Hayley Dorfman
• Morgan Glover
• Daphne Valencia
• Xinxu Shen
• Samuel Gershman
• Catherine Hartley

The ability to learn from positive and negative outcomes is essential throughout the lifespan. Previous research in adults has shown that valence-dependent learning rates can be modulated by beliefs about the causal structure of the environment. The present study examined whether causal judgments simi- larly influence learning across development. Participants completed a reinforcement learning task in which they had to choose between two options with fixed reward probabilities. Participants made choices across three distinct environments. In each environment, a different hidden agent occasionally intervened to gener- ate positive, negative, or random outcomes. This manipulation has been previously shown to bias learning rates, such that participants update their value estimates of each option to a lesser extent when the outcomes of their choices can be attributed to the agent (Dorfman, Bhui, Hughes, & Gershman, 2019). Analyses of data from 88 individuals ages 7 to 25 show that participants’ beliefs about hidden agent intervention align with the manipulation of positive, negative, or random outcomes in each of the three environments. Compu- tational modeling of the learning data revealed that while the choices made by both adults (ages 18 - 25) and adolescents (ages 13 - 17) are best fit by a Bayesian reinforcement learning model that incorporates beliefs about hidden agent intervention, those of children are best fit by a two-learning-rate model that updates value estimates based on choice outcomes alone. Together, these results suggest that while children demonstrate explicit awareness of the causal structure of the task environment, unlike adolescents and adults, they do not implicitly use beliefs about the causal structure of the environment to guide reinforcement learning.

RLDM Conference 2019 Conference Abstract

Model-free and model-based learning processes in the updating of explicit and implicit evaluations

• Benedek Kurdi
• Samuel Gershman

Evaluating stimuli along a positive–negative dimension is a fundamental computation performed by the human mind. In recent decades, research has documented both dissociations and associations be- tween explicit (self-reported) and implicit (indirectly measured) forms of evaluations. Together, these two forms of evaluation are central to organizing social cognition and drive behavior in intergroup relations, consumer choice, psychopathology, and close relationships. However, it is unclear whether explicit–implicit dissociations arise from relatively more superficial differences in measurement techniques or from deeper differences in the processes by which explicit and implicit evaluations are acquired and represented. The cur- rent project (total sample size: N = 2, 354) relies on the computationally well-specified distinction between model-based and model-free reinforcement learning to investigate the unique and shared aspects of explicit and implicit evaluations. Study 1 used a revaluation procedure to reveal that whereas explicit evaluations of novel targets are updated via both model-free and model-based processes, implicit evaluations depend on the former but are impervious to the latter. Studies 2–3 demonstrated the robustness of this effect to (a) the number of stimulus exposures in the revaluation phase and (b) the deterministic vs. probabilistic nature of initial reinforcement. These findings provide a novel framework, going beyond traditional dual-process and single-process accounts, to highlight the context-sensitivity and long-term recalcitrance of implicit evalua- tions as well as variations in their relationship with their explicit counterparts. These results also suggest novel avenues for designing theoretically guided interventions to produce change in implicit evaluations.

NeurIPS Conference 2018 Conference Paper

Human-in-the-Loop Interpretability Prior

• Isaac Lage
• Andrew Ross
• Samuel Gershman
• Been Kim
• Finale Doshi-Velez

We often desire our models to be interpretable as well as accurate. Prior work on optimizing models for interpretability has relied on easy-to-quantify proxies for interpretability, such as sparsity or the number of operations required. In this work, we optimize for interpretability by directly including humans in the optimization loop. We develop an algorithm that minimizes the number of user studies to find models that are both predictive and interpretable and demonstrate our approach on several data sets. Our human subjects results show trends towards different proxy notions of interpretability on different datasets, which suggests that different proxies are preferred on different tasks.

RLDM Conference 2017 Conference Abstract

Metacontrol in reinforcement learning

• Wouter Kool
• Fiery Cushman
• Samuel Gershman

Decision making is sometimes guided by habit, and at other times by goal-directed planning. This distinction has recently been formalized as a competition between a computationally cheap but inflexible ”model-free” system, and an expensive but more flexible ”model-based” system. This formalization has facilitated our understanding of the neural and cognitive mechanisms underlying these decision-making systems, but it remains unclear how the brain allocates control between them. Here, we present behavioral, computational, and neuroimaging analyses that converge on a common conclusion: Arbitration is guided by a comparison of each systems benefits, discounted by a cost for model-based control. First, we describe a new sequential decision task that can dissociate between model-free and model-based control. In contrast to prior tasks, enhanced model-based control on our task yields increased reward. Next, we used a stakes manipulation to capitalize on this difference between tasks to test our cost-benefit account. On certain trials, a cue signaled that the rewards would be amplified. Consistent with the cost-benefit hypothesis, we found increased reliance on model-based control on high-stakes trials, but only in our task, where high stakes enhance the benefits of planning. We account for these findings with a reinforcement learning model that adaptively arbitrates between model-based and model-free control using a policy gradient algorithm. This allocation is guided by ’controller values’ which integrate the costs and benefits of using each controller, and are updated according to ’controller prediction errors. ’ This model provides a superior behavioral fit compared to previous models. A neuroimaging study provides convergent evidence for this model, revealing a set of regions in frontal cortex, commonly associated with valuation and cognitive control, which encode the controller prediction error.

NeurIPS Conference 2016 Conference Paper

Probing the Compositionality of Intuitive Functions

• Eric Schulz
• Josh Tenenbaum
• David Duvenaud
• Maarten Speekenbrink
• Samuel Gershman

How do people learn about complex functional structure? Taking inspiration from other areas of cognitive science, we propose that this is accomplished by harnessing compositionality: complex structure is decomposed into simpler building blocks. We formalize this idea within the framework of Bayesian regression using a grammar over Gaussian process kernels. We show that participants prefer compositional over non-compositional function extrapolations, that samples from the human prior over functions are best described by a compositional model, and that people perceive compositional functions as more predictable than their non-compositional but otherwise similar counterparts. We argue that the compositional nature of intuitive functions is consistent with broad principles of human cognition.

RLDM Conference 2015 Conference Abstract

The successor representation in human reinforcement learning: evidence from retrospective revaluation

• Ida Momennejad
• Jin Cheong
• Matthew Botvinick
• Samuel Gershman

Reinforcement learning (RL) has been posed as a competition between model-free (MF) and model-based (MB) learning. MF learning cannot solve problems such as revaluation and latent learning, hallmarks of MB behavior. However, we suggest that there are varieties of MB-like behavior that are also predicted by an alternative solution to the RL problem that lies between the MF and MB strategies. In particular, the successor representation (SR) can account for certain kinds of retrospective revaluation of rewards, a behavior that has traditionally been ascribed to the MB system. We conducted two experiments to test this hypothesis and compared the classic ‘reward devaluation’ (reward structure changes, transition structure stays the same) with ‘transition devaluation’ (reward structure stays the same, transition structure changes). A pure SR strategy will only be sensitive to reward but not transition devaluation, because the SR effectively ‘compiles’ the transition structure and therefore cannot adapt quickly to changes. Behav- ioral results from Study 1 showed that subjects were more sensitive to reward than transition devaluation (n=58, p<. 05). However, subjects still showed some sensitivity to transition devaluation, inconsistent with a pure SR strategy. These results point to the possibility that subjects may employ a mixed SR-MB strategy, whereby the value function for a MB strategy is initialized using the SR. With more processing time, the in- fluence of the initialization diminishes, causing behavior to resemble a pure MB strategy. Study 2 tested the hybrid SR-MB model of retrospective revaluation. Consistent with our predictions, fast responses showed greater sensitivity to reward than transition devaluation, while slower responses displayed equal sensitivity to both (n=52, p<. 05). Very slow responses showed low sensitivity to both types of devaluation; consistent with the hypothesis that noise accumulates in the MB computation, impairing performance.

NeurIPS Conference 2014 Conference Paper

Design Principles of the Hippocampal Cognitive Map

• Kimberly Stachenfeld
• Matthew Botvinick
• Samuel Gershman

Hippocampal place fields have been shown to reflect behaviorally relevant aspects of space. For instance, place fields tend to be skewed along commonly traveled directions, they cluster around rewarded locations, and they are constrained by the geometric structure of the environment. We hypothesize a set of design principles for the hippocampal cognitive map that explain how place fields represent space in a way that facilitates navigation and reinforcement learning. In particular, we suggest that place fields encode not just information about the current location, but also predictions about future locations under the current transition distribution. Under this model, a variety of place field phenomena arise naturally from the structure of rewards, barriers, and directional biases as reflected in the transition policy. Furthermore, we demonstrate that this representation of space can support efficient reinforcement learning. We also propose that grid cells compute the eigendecomposition of place fields in part because is useful for segmenting an enclosure along natural boundaries. When applied recursively, this segmentation can be used to discover a hierarchical decomposition of space. Thus, grid cells might be involved in computing subgoals for hierarchical reinforcement learning.

RLDM Conference 2013 Conference Abstract

Hierarchical deconstruction and memoization of goal-directed plans

• Quentin Huys
• Niall Lally
• Paul Falkner
• Samuel Gershman
• Jonathan Roiser

Humans cannot exactly solve most planning problems they face, but must approximate them. Research has characterized one class of approximations, whereby experience accumulated in habits substi- tutes for the computational expense of searching goal-directed decision-trees. Building on previous work in which we characterised Pavlovian pruning of decision trees, we explore more efficient approximations to the planning problem. We focus on habit-like caching (’memoization’) of more complex action sequences in dynamic, hierarchical decompositions of complex decision-trees. Three groups of subjects performed a planning task. They first learned a transition matrix and then learned about rewards associated with the transitions. They then produced choice sequences of a given length from a random starting state to maximise their total earnings. Using reinforcement learning models nested inside a Chinese Restaurant Process we infer subject’s hierarchical decomposition, stochastic memoization and pruning strategies. Hierarchical decomposition and stochastic memoization models give detailed accounts of complex features of choice data. We characterise how subjects dynamically establish subgoals; that their decomposition strategy achieves a near optimal trade-off between computational costs and gains; that sub- jects memoized and re-used complex choice sequences; that this correlated negatively with their ability to search the tree; and replicated our previous findings whereby subjects disregard (prune) subtrees that lie below large losses. We replicate all findings in two further datasets. Humans employ multiple approximations when solving complex planning problems. They simplify the problem by establishing sub-goals and decomposing the decision-tree around these in a manner that trades computational costs for gains near optimally. They do not re-compute solutions on every trial but rather re-use previous solutions.

ICML Conference 2012 Conference Paper

Nonparametric variational inference

• Samuel Gershman
• Matthew D. Hoffman
• David M. Blei

NeurIPS Conference 2010 Conference Paper

The Neural Costs of Optimal Control

• Samuel Gershman
• Robert Wilson

Optimal control entails combining probabilities and utilities. However, for most practical problems probability densities can be represented only approximately. Choosing an approximation requires balancing the benefits of an accurate approximation against the costs of computing it. We propose a variational framework for achieving this balance and apply it to the problem of how a population code should optimally represent a distribution under resource constraints. The essence of our analysis is the conjecture that population codes are organized to maximize a lower bound on the log expected utility. This theory can account for a plethora of experimental data, including the reward-modulation of sensory receptive fields.

NeurIPS Conference 2009 Conference Paper

A Bayesian Analysis of Dynamics in Free Recall

• Richard Socher
• Samuel Gershman
• Per Sederberg
• Kenneth Norman
• Adler Perotte
• David Blei

We develop a probabilistic model of human memory performance in free recall experiments. In these experiments, a subject first studies a list of words and then tries to recall them. To model these data, we draw on both previous psychological research and statistical topic models of text documents. We assume that memories are formed by assimilating the semantic meaning of studied words (represented as a distribution over topics) into a slowly changing latent context (represented in the same space). During recall, this context is reinstated and used as a cue for retrieving studied words. By conceptualizing memory retrieval as a dynamic latent variable model, we are able to use Bayesian inference to represent uncertainty and reason about the cognitive processes underlying memory. We present a particle filter algorithm for performing approximate posterior inference, and evaluate our model on the prediction of recalled words in experimental data. By specifying the model hierarchically, we are also able to capture inter-subject variability.

NeurIPS Conference 2009 Conference Paper

Perceptual Multistability as Markov Chain Monte Carlo Inference

• Samuel Gershman
• Ed Vul
• Joshua Tenenbaum

While many perceptual and cognitive phenomena are well described in terms of Bayesian inference, the necessary computations are intractable at the scale of real-world tasks, and it remains unclear how the human mind approximates Bayesian inference algorithmically. We explore the proposal that for some tasks, humans use a form of Markov Chain Monte Carlo to approximate the posterior distribution over hidden variables. As a case study, we show how several phenomena of perceptual multistability can be explained as MCMC inference in simple graphical models for low-level vision.