Author name cluster

Özgür Şimşek

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

11 papers

1 author row

NeurIPS Conference 2025 Conference Paper

Approximating Shapley Explanations in Reinforcement Learning

Daniel Beechey
Özgür Şimşek

Reinforcement learning has achieved remarkable success in complex decision-making environments, yet its lack of transparency limits its deployment in practice, especially in safety-critical settings. Shapley values from cooperative game theory provide a principled framework for explaining reinforcement learning; however, the computational cost of Shapley explanations is an obstacle for their use. We introduce FastSVERL, a scalable method for explaining reinforcement learning by approximating Shapley values. FastSVERL is designed to handle the unique challenges of reinforcement learning, including temporal dependencies across multi-step trajectories, learning from off-policy data, and adapting to evolving agent behaviours in real time. FastSVERL introduces a practical, scalable approach for principled and rigourous interpretability in reinforcement learning.

PDF Details

EWRL Workshop 2025 Workshop Paper

Learning the Minimum Action Distance

Lorenzo Steccanella
Joshua Benjamin Evans
Özgür Şimşek
Anders Jonsson

This paper presents a state representation framework for Markov decision processes (MDPs) that can be learned solely from state trajectories, requiring neither reward signals nor the actions executed by the agent. We propose learning the $\textit{minimum action distance}$ (MAD), defined as the minimum number of actions required to transition between states, as a fundamental metric that captures the underlying structure of an environment. MAD naturally enables critical downstream tasks such as goal-conditioned reinforcement learning and reward shaping by providing a dense, geometrically meaningful measure of progress. Our self-supervised learning approach constructs an embedding space where the distances between embedded state pairs correspond to their MAD, accommodating both symmetric and asymmetric approximations. We evaluate the framework on a comprehensive suite of environments with known MAD values, encompassing both deterministic and stochastic dynamics, as well as discrete and continuous state spaces, and environments with noisy observations. Empirical results demonstrate that the proposed approach not only efficiently learns accurate MAD representations across these diverse settings but also significantly outperforms existing state representation methods in terms of representation quality.

PDF

NeurIPS Conference 2025 Conference Paper

Novel Exploration via Orthogonality

Andreas Theophilou
Özgür Şimşek

Efficient exploration remains one of the most important open problems in reinforcement learning. Discovering novel states or transitions requires policies that efficiently direct the agent away from the regions of the state space that are already well explored. We introduce Novel Exploration via Orthogonality (NEO), an approach that automatically uncovers not only which regions of the environment are novel but also how to reach them by leveraging Laplacian representations. NEO uses the eigenvectors of a modified graph Laplacian to induce gradient flows from states that are frequently visited (less novel) to states that are seldom visited (more novel). We show that NEO's modified Laplacian yields eigenvectors whose extreme values align with the most novel regions of the state space. We provide bounds for the eigenvalues of the modified Laplacian; and we show that the smoothest eigenvectors with real eigenvalues below certain thresholds provide guaranteed gradients to novel states for both undirected and directed graphs. In an empirical evaluation in online, incremental settings, NEO outperformed related state-of-the-art approaches, including eigen-options and cover options, in a large collection of undirected and directed environments with varying connectivity structures.

PDF Details

EWRL Workshop 2024 Workshop Paper

Curricula for Learning Robust Policies with Factored State Representations in Changing Environments

Panayiotis Panayiotou
Özgür Şimşek

Robust policies enable reinforcement learning agents to effectively adapt to and operate in unpredictable, dynamic, and ever-changing real-world environments. Factored representations, which break down complex state and action spaces into distinct components, can improve generalization and sample efficiency in policy learning. In this paper, we explore how the curriculum of an agent using a factored state representation affects the robustness of the learned policy. We experimentally demonstrate three simple curricula, such as varying only the variable of highest regret between episodes, that can significantly enhance policy robustness, offering practical insights for reinforcement learning in complex environments.

PDF

EWRL Workshop 2024 Workshop Paper

Interpreting Reinforcement Learning with Shapley Values

Daniel Beechey
Thomas M. S. Smith
Özgür Şimşek

For reinforcement learning systems to be widely adopted, their users must understand and trust them. We present a theoretical analysis of explaining reinforcement learning using Shapley values, following a principled approach from game theory for identifying the contribution of individual players to the outcome of a cooperative game. We call this general framework Shapley Values for Explaining Reinforcement Learning (SVERL). Our analysis exposes the limitations of earlier uses of Shapley values in reinforcement learning. We then develop an approach that uses Shapley values to explain agent performance. In a variety of domains, SVERL produces meaningful explanations that match and supplement human intuition.

PDF

EWRL Workshop 2024 Workshop Paper

Louvain Skills: Building Multi-Level Skill Hierarchies in Reinforcement Learning

Joshua Benjamin Evans
Özgür Şimşek

What is a useful skill hierarchy for an autonomous agent? We propose an answer based on a graphical representation of how the interaction between an agent and its environment may unfold. Our approach uses modularity maximisation as a central organising principle to expose the structure of the interaction graph at multiple levels of abstraction. The result is a collection of skills that operate at varying time scales, organised into a hierarchy, where skills that operate over longer time scales are composed of skills that operate over shorter time scales. The entire skill hierarchy is generated automatically, with no human intervention, including the skills themselves (their behaviour, when they can be called, and when they terminate) as well as the hierarchical dependency structure between them. In a wide range of environments, this approach generates skill hierarchies that are intuitively appealing and that considerably improve the learning performance of the agent.

PDF

YNIMG Journal 2024 Journal Article

Solving large-scale MEG/EEG source localisation and functional connectivity problems simultaneously using state-space models

Jose Sanchez-Bornot
Roberto C. Sotero
J.A. Scott Kelso
Özgür Şimşek
Damien Coyle

State-space models are widely employed across various research disciplines to study unobserved dynamics. Conventional estimation techniques, such as Kalman filtering and expectation maximisation, offer valuable insights but incur high computational costs in large-scale analyses. Sparse inverse covariance estimators can mitigate these costs, but at the expense of a trade-off between enforced sparsity and increased estimation bias, necessitating careful assessment in low signal-to-noise ratio (SNR) situations. To address these challenges, we propose a three-fold solution: (1) Introducing multiple penalised state-space (MPSS) models that leverage data-driven regularisation; (2) Developing novel algorithms derived from backpropagation, gradient descent, and alternating least squares to solve MPSS models; (3) Presenting a K-fold cross-validation extension for evaluating regularisation parameters. We validate this MPSS regularisation framework through lower and more complex simulations under varying SNR conditions, including a large-scale synthetic magneto- and electro-encephalography (MEG/EEG) data analysis. In addition, we apply MPSS models to concurrently solve brain source localisation and functional connectivity problems for real event-related MEG/EEG data, encompassing thousands of sources on the cortical surface. The proposed methodology overcomes the limitations of existing approaches, such as constraints to small-scale and region-of-interest analyses. Thus, it may enable a more accurate and detailed exploration of cognitive brain functions.

Details DOI

NeurIPS Conference 2023 Conference Paper

Creating Multi-Level Skill Hierarchies in Reinforcement Learning

Joshua B. Evans
Özgür Şimşek

What is a useful skill hierarchy for an autonomous agent? We propose an answer based on a graphical representation of how the interaction between an agent and its environment may unfold. Our approach uses modularity maximisation as a central organising principle to expose the structure of the interaction graph at multiple levels of abstraction. The result is a collection of skills that operate at varying time scales, organised into a hierarchy, where skills that operate over longer time scales are composed of skills that operate over shorter time scales. The entire skill hierarchy is generated automatically, with no human input, including the skills themselves (their behaviour, when they can be called, and when they terminate) as well as the dependency structure between them. In a wide range of environments, this approach generates skill hierarchies that are intuitively appealing and that considerably improve the learning performance of the agent.

PDF Details

NeurIPS Conference 2015 Conference Paper

Learning From Small Samples: An Analysis of Simple Decision Heuristics

Özgür Şimşek
Marcus Buckmann

Simple decision heuristics are models of human and animal behavior that use few pieces of information---perhaps only a single piece of information---and integrate the pieces in simple ways, for example, by considering them sequentially, one at a time, or by giving them equal weight. It is unknown how quickly these heuristics can be learned from experience. We show, analytically and empirically, that only a few training samples lead to substantial progress in learning. We focus on three families of heuristics: single-cue decision making, lexicographic decision making, and tallying. Our empirical analysis is the most extensive to date, employing 63 natural data sets on diverse subjects.

PDF Details

NeurIPS Conference 2013 Conference Paper

Linear decision rule as aspiration for simple decision heuristics

Özgür Şimşek

Many attempts to understand the success of simple decision heuristics have examined heuristics as an approximation to a linear decision rule. This research has identified three environmental structures that aid heuristics: dominance, cumulative dominance, and noncompensatoriness. Here, we further develop these ideas and examine their empirical relevance in 51 natural environments. We find that all three structures are prevalent, making it possible for some simple rules to reach the accuracy levels of the linear decision rule using less information.

PDF Details

NeurIPS Conference 2008 Conference Paper

Skill Characterization Based on Betweenness

Özgür Şimşek
Andrew Barto

We present a characterization of a useful class of skills based on a graphical representation of an agent's interaction with its environment. Our characterization uses betweenness, a measure of centrality on graphs. It may be used directly to form a set of skills suitable for a given environment. More importantly, it serves as a useful guide for developing online, incremental skill discovery algorithms that do not rely on knowing or representing the environment graph in its entirety.

PDF Details