Arrow Research search

Author name cluster

Sam Lobel

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

9 papers
2 author rows

Possible papers

9

RLJ Journal 2025 Journal Article

Intrinsically Motivated Discovery of Temporally Abstract Graph-based Models of the World

  • Akhil Bagaria
  • Anita De Mello Koch
  • Rafael Rodriguez-Sanchez
  • Sam Lobel
  • George Konidaris

We seek to design reinforcement learning agents that build plannable models of the world that are abstract in both state and time. We propose a new algorithm to construct a skill graph; nodes in the skill graph represent abstract states and edges represent skill policies. Previous works that learn a skill graph use random sampling from the state-space and nearest-neighbor search: operations that are infeasible in environments with high-dimensional observations (for example, images). Furthermore, previous algorithms attempt to increase the probability of all edges (by repeatedly executing the corresponding skills) so that the resulting graph is robust and reliable everywhere. However, exhaustive coverage is infeasible in large environments, and agents should prioritize practicing skills that are more likely to result in higher reward. We show that our agent can solve challenging image-based exploration problems more rapidly than vanilla model-free RL and state-of-the-art novelty-based exploration; then, we show that the resulting abstract model solve a family of tasks not provided during the agent's exploration phase.

PDF Details

RLC Conference 2025 Conference Paper

Intrinsically Motivated Discovery of Temporally Abstract Graph-based Models of the World

  • Akhil Bagaria
  • Anita De Mello Koch
  • Rafael Rodriguez-Sanchez
  • Sam Lobel
  • George Konidaris

We seek to design reinforcement learning agents that build plannable models of the world that are abstract in both state and time. We propose a new algorithm to construct a skill graph; nodes in the skill graph represent abstract states and edges represent skill policies. Previous works that learn a skill graph use random sampling from the state-space and nearest-neighbor search: operations that are infeasible in environments with high-dimensional observations (for example, images). Furthermore, previous algorithms attempt to increase the probability of all edges (by repeatedly executing the corresponding skills) so that the resulting graph is robust and reliable everywhere. However, exhaustive coverage is infeasible in large environments, and agents should prioritize practicing skills that are more likely to result in higher reward. We show that our agent can solve challenging image-based exploration problems more rapidly than vanilla model-free RL and state-of-the-art novelty-based exploration; then, we show that the resulting abstract model solve a family of tasks not provided during the agent's exploration phase.

PDF Details

NeurIPS Conference 2025 Conference Paper

Skill-Driven Neurosymbolic State Abstractions

  • Alper Ahmetoglu
  • Steven James
  • Cameron Allen
  • Sam Lobel
  • David Abel
  • George Konidaris

We consider how to construct state abstractions compatible with a given set of abstract actions, to obtain a well-formed abstract Markov decision process (MDP). We show that the Bellman equation suggests that abstract states should represent distributions over states in the ground MDP; we characterize the conditions under which the resulting process is Markov and approximately model-preserving, derive algorithms for constructing and planning with the abstract MDP, and apply them to a visual maze task. We generalize these results to the factored actions case, characterizing the conditions that result in factored abstract states and apply the resulting algorithm to Montezuma's Revenge. These results provide a powerful and principled framework for constructing neurosymbolic abstract Markov decision processes.

PDF Details

RLC Conference 2024 Conference Paper

An Optimal Tightness Bound for the Simulation Lemma

  • Sam Lobel
  • Ronald Parr

We present a bound for value-prediction error with respect to model misspecification that is tight, including constant factors. This is a direct improvement of the ``simulation lemma, ’’ a foundational result in reinforcement learning. We demonstrate that existing bounds are quite loose, becoming vacuous for large discount factors, due to the suboptimal treatment of compounding probability errors. By carefully considering this quantity on its own, instead of as a subcomponent of value error, we derive a bound that is sub-linear with respect to transition function misspecification. We then demonstrate broader applicability of this technique, improving a similar bound in the related subfield of hierarchical abstraction.

PDF Details

RLJ Journal 2024 Journal Article

An Optimal Tightness Bound for the Simulation Lemma

  • Sam Lobel
  • Ronald Parr

We present a bound for value-prediction error with respect to model misspecification that is tight, including constant factors. This is a direct improvement of the ``simulation lemma,’’ a foundational result in reinforcement learning. We demonstrate that existing bounds are quite loose, becoming vacuous for large discount factors, due to the suboptimal treatment of compounding probability errors. By carefully considering this quantity on its own, instead of as a subcomponent of value error, we derive a bound that is sub-linear with respect to transition function misspecification. We then demonstrate broader applicability of this technique, improving a similar bound in the related subfield of hierarchical abstraction.

PDF Details

NeurIPS Conference 2024 Conference Paper

Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy

  • Cameron Allen
  • Aaron Kirtland
  • Ruo Yu Tao
  • Sam Lobel
  • Daniel Scott
  • Nicholas Petrocelli
  • Omer Gottesman
  • Ronald Parr

Reinforcement learning algorithms typically rely on the assumption that the environment dynamics and value function can be expressed in terms of a Markovian state representation. However, when state information is only partially observable, how can an agent learn such a state representation, and how can it detect when it has found one? We introduce a metric that can accomplish both objectives, without requiring access to---or knowledge of---an underlying, unobservable state space. Our metric, the λ-discrepancy, is the difference between two distinct temporal difference (TD) value estimates, each computed using TD(λ) with a different value of λ. Since TD(λ=0) makes an implicit Markov assumption and TD(λ=1) does not, a discrepancy between these estimates is a potential indicator of a non-Markovian state representation. Indeed, we prove that the λ-discrepancy is exactly zero for all Markov decision processes and almost always non-zero for a broad class of partially observable environments. We also demonstrate empirically that, once detected, minimizing the λ-discrepancy can help with learning a memory function to mitigate the corresponding partial observability. We then train a reinforcement learning agent that simultaneously constructs two recurrent value networks with different λ parameters and minimizes the difference between them as an auxiliary loss. The approach scales to challenging partially observable domains, where the resulting agent frequently performs significantly better (and never performs worse) than a baseline recurrent agent with only a single value network.

PDF Details DOI

ICML Conference 2023 Conference Paper

Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learning

  • Sam Lobel
  • Akhil Bagaria
  • George Konidaris 0001

We propose a new method for count-based exploration in high-dimensional state spaces. Unlike previous work which relies on density models, we show that counts can be derived by averaging samples from the Rademacher distribution (or coin flips). This insight is used to set up a simple supervised learning objective which, when optimized, yields a state’s visitation count. We show that our method is significantly more effective at deducing ground-truth visitation counts than previous work; when used as an exploration bonus for a model-free reinforcement learning algorithm, it outperforms existing approaches on most of 9 challenging exploration tasks, including the Atari game Montezuma’s Revenge.

Details

AAAI Conference 2022 Conference Paper

Optimistic Initialization for Exploration in Continuous Control

  • Sam Lobel
  • Omer Gottesman
  • Cameron Allen
  • Akhil Bagaria
  • George Konidaris

Optimistic initialization underpins many theoretically sound exploration schemes in tabular domains; however, in the deep function approximation setting, optimism can quickly disappear if initialized naı̈vely. We propose a framework for more effectively incorporating optimistic initialization into reinforcement learning for continuous control. Our approach uses metric information about the state-action space to estimate which transitions are still unexplored, and explicitly maintains the initial Q-value optimism for the corresponding state-action pairs. We also develop methods for efficiently approximating these training objectives, and for incorporating domain knowledge into the optimistic envelope to improve sample efficiency. We empirically evaluate these approaches on a variety of hard exploration problems in continuous control, where our method outperforms existing exploration techniques.

PDF Details

ICLR Conference 2020 Conference Paper

RaCT: Toward Amortized Ranking-Critical Training For Collaborative Filtering

  • Sam Lobel
  • Chunyuan Li
  • Jianfeng Gao 0001
  • Lawrence Carin

We investigate new methods for training collaborative filtering models based on actor-critic reinforcement learning, to more directly maximize ranking-based objective functions. Specifically, we train a critic network to approximate ranking-based metrics, and then update the actor network to directly optimize against the learned metrics. In contrast to traditional learning-to-rank methods that require re-running the optimization procedure for new lists, our critic-based method amortizes the scoring process with a neural network, and can directly provide the (approximate) ranking scores for new lists. We demonstrate the actor-critic's ability to significantly improve the performance of a variety of prediction models, and achieve better or comparable performance to a variety of strong baselines on three large-scale datasets.

Details
v2026.09.13