Author name cluster

Samuele Tosatto

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

12 papers

2 author rows

AAAI Conference 2026 Conference Paper

Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarks

Muthukumar Pandaram
Jakob Hollenstein
David Drexel
Samuele Tosatto
Antonio Rodriguez-Sanchez
Justus Piater

The use of learned dynamics models, also known as world models, can improve the sample efficiency of reinforcement learning. Recent work suggests that the underlying causal graphs of such dynamics models are sparsely connected, with each of the future state variables depending only on a small subset of the current state variables, and that learning may therefore benefit from sparsity priors. Similarly, temporal sparsity, i.e. sparsely and abruptly changing local dynamics, has also been proposed as a useful inductive bias. In this work, we critically examine these assumptions by analyzing ground truth dynamics from a set of robotic reinforcement learning environments in the MuJoCo Playground benchmark suite, aiming to determine whether the proposed notions of state and temporal sparsity actually tend to hold in typical reinforcement learning tasks. We study (i) whether the causal graphs of environment dynamics are sparse, (ii) whether such sparsity is state-dependent, and (iii) whether local system dynamics change sparsely. Our results indicate that global sparsity is rare, but instead the tasks show local, state-dependent sparsity in their dynamics and this sparsity exhibits distinct structures, appearing in temporally localized clusters (e.g., during contact events) and affecting specific subsets of state dimensions. These findings challenge common sparsity prior assumptions in dynamics learning, emphasizing the need for grounded inductive biases that reflect the state-dependent sparsity structure of real-world dynamics.

PDF Details DOI

EWRL Workshop 2025 Workshop Paper

Pink Noise LQR: How does Colored Noise affect the Optimal Policy in RL?

Jakob Hollenstein
Marko Zaric
Samuele Tosatto
Justus Piater

Colored noise, a class of temporally correlated noise processes, has shown promising results for improving exploration in deep reinforcement learning for both off-policy and on-policy algorithms. However, it is unclear how temporally correlated colored noise affects policy learning apart from changing exploration properties. In this paper, we investigate the implications of colored noise on on-policy deep reinforcement learning in a simplified setting, considering linear dynamics and a linear policy under quadratic costs. We derive a closed-form solution for the expected cost, revealing that colored noise affects both the expected cost and the optimal policy. Notably, the cost splits into a state-cost part equal to the unperturbed system’s cost and a noise-cost term, affecting the policy, but independent of the initial state. While the cost changes depending on the noise, the expected trajectory remains independent of the noise color for a given linear policy. Far from the goal state, the state cost dominates, and the effect due to the noise is negligible: the policy approaches the optimal policy of the unperturbed system. Near the goal state, the noise cost dominates, changing the optimal policy.

PDF

EWRL Workshop 2025 Workshop Paper

Revisiting Proximal Policy Optimization

Mahdi Kallel
Jose-Luis Holgado-Alvarez
Samuele Tosatto
Carlo D'Eramo

On-policy Reinforcement Learning (RL) offers desirable features such as stable learning, fewer policy updates, and the ability to evaluate a policy’s return during training. While recent efforts have focused on off-policy methods, achieving significant advancements, PPO remains the go-to algorithm for on-policy RL due to its apparent simplicity and effectiveness. Nonetheless, PPO is highly sensitive to hyperparameters and relies on subtle, often poorly documented adjustments that can critically affect its performance, thereby limiting its utility in complex scenarios. In this paper, we revisit the PPO algorithm by introducing principled enhancements that improve performance while eliminating the need for extensive hyperparameter tuning and implementation-specific optimizations. Our proposed approach, PPO+, is a principled adaptation of the PPO algorithm that strengthens its adherence to the on-policy objective, enhancing stability and efficiency. PPO+ sets a new state-of-the-art for deep on-policy RL on MuJoCo control problems while maintaining a straightforward implementation. PPO+ demonstrates significantly improved asymptotic performance over PPO, and a substantially reduced performance gap with off-policy algorithms in several challenging continuous control tasks. Beyond just performance, our findings offer a fresh perspective on on-policy RL.

PDF

IROS Conference 2024 Conference Paper

Local Linearity is All You Need (in Data-Driven Teleoperation)

Michael Przystupa
Gauthier Gidel
Matthew E. Taylor
Martin Jägersand
Justus H. Piater
Samuele Tosatto

One of the critical aspects of assistive robotics is to provide a control system of a high-dimensional robot from a low-dimensional user input (i. e. a 2D joystick). Data-driven teleoperation seeks to provide an intuitive user interface called an action map to map the low dimensional input to robot velocities from human demonstrations. Action maps are machine learning models trained on robotic demonstration data to map user input directly to desired movements as opposed to aspects of robot pose ("move to cup or pour content" vs. "move along x- or y-axis"). Many works have investigated nonlinear action maps with multi-layer perceptrons, but recent work suggests that local-linear neural approximations provide better control of the system. However, local linear models assume actions exist on a linear subspace and may not capture nuanced motions in training data. In this work, we hypothesize that local-linear neural networks are effective because they make the action map odd w. r. t. the user input, enhancing the intuitiveness of the controller. Based on this assumption, we propose two nonlinear means of encoding odd behavior that do not constrain the action map to a local linear function. However, our analysis reveals that these models effectively behave like local linear models for relevant mappings between user joysticks and robot movements. We support this claim in simulation, and show on a realworld use case that there is no statistical benefit of using non-linear maps, according to the users experience. These negative results suggest that further investigation into model architectures beyond local linear models may offer diminishing returns for improving user experience in data-driven teleoperation systems.

Details

EWRL Workshop 2024 Workshop Paper

Revisiting On-Policy Deep Reinforcement Learning

Mahdi Kallel
Samuele Tosatto
Carlo D'Eramo

On-policy Reinforcement Learning (RL) offers several desirable properties, including more stable learning, less frequent policy changes, and the capacity to evaluate a policy's return during the learning process. Despite the considerable success of recent off-policy methods, their on-policy counterparts continue to lag in terms of asymptotic performance and sample efficiency. Proximal Policy Optimization (PPO) remains the de facto standard, despite its complexity and demonstrated sensitivity to hyperparameters. In this work, we introduce On-Policy Soft Actor-Critic (ON-SAC), a methodical adaptation of the Soft Actor-Critic (SAC) algorithm tailored for the on-policy setting. Our approach begins with the observation that the current on-policy algorithms do not use true on-policy gradients. We build on this observation to offer founded remedies for this problem. Our algorithm establishes a new state-of-the-art for deep on-policy RL, while simplifying the process by eliminating the need for trust-region methods and intricate critic learning schemes.

PDF

EWRL Workshop 2023 Workshop Paper

A Gradient Critic for Policy Gradient Estimation

Samuele Tosatto
Andrew Patterson
Martha White
A. Rupam Mahmood

The policy gradient theorem (Sutton et al. , 2000) prescribes the usage of the on-policy state distribution to approximate the gradient. Most algorithms based on this theorem, in practice, break this assumption introducing a distribution shift that can cause the convergence to poor solutions. In this paper, we propose a new approach of reconstructing the policy gradient from the start state without requiring a particular sampling strategy. The policy gradient calculation in this form can be simplified in terms of a \textsl{gradient critic}, which can be recursively estimated due to a new Bellman equation of gradients. By using temporal-difference updates of the gradient critic from an off-policy data stream, we develop the first estimator that side-steps the distribution shift issue in a model-free way. We prove that, under certain realizability conditions, our estimator is unbiased regardless of the sampling strategy. We empirically show that our technique achieves a superior bias-variance trade-off and performance in the presence of off-policy samples. The extended version of this work can be found in Tosatto et al. (2022), and the implementation of the experiment at github. com/SamuelePolimi/temporal-difference-gradient.

PDF

IROS Conference 2023 Conference Paper

Deep Probabilistic Movement Primitives with a Bayesian Aggregator

Michael Przystupa
Faezeh Haghverd
Martin Jägersand
Samuele Tosatto

Movement primitives are trainable parametric models that reproduce robotic movements starting from a limited set of demonstrations. Previous works proposed simple linear models that exhibited high sample efficiency and generalization power by allowing temporal modulation of move-ments (reproducing movements faster or slower), blending (merging two movements into one), via-point conditioning (constraining a movement to meet some particular via-points) and context conditioning (generation of movements based on an observed variable, e. g. , position of an object). Previous works have proposed neural network-based motor primitive models, having demonstrated their capacity to perform tasks with some forms of input conditioning or time-modulation representations. However, there has not been a single unified deep movement primitive's model proposed that is capable of all previous operations, limiting neural movement primitive's potential applications. This paper proposes a deep movement primitive architecture that encodes all the operations above and uses a Bayesian context aggregator that allows a more sound context conditioning and blending. Our results demonstrate our approach can scale to reproduce complex motions on a larger variety of input choices compared to baselines while maintaining operations of linear movement primitives provide.

Details

IROS Conference 2023 Conference Paper

Dynamic Decision Frequency with Continuous Options

Amirmohammad Karimi
Jun Jin 0001
Jun Luo 0009
A. Rupam Mahmood
Martin Jägersand
Samuele Tosatto

In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The duration between decisions becomes a crucial hyperparameter, as setting it too short may increase the problem's difficulty by requiring the agent to make numerous decisions to achieve its goal while setting it too long can result in the agent losing control over the system. However, physical systems do not necessarily require a constant control frequency, and for learning agents, it is often preferable to operate with a low frequency when possible and a high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. These options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. We demonstrate the effectiveness of CTCO by comparing its performance to classical RL and temporal-abstraction RL methods on simulated continuous control tasks with various action-cycle times. We show that our algorithm's performance is not affected by the choice of environment interaction frequency. Furthermore, we demonstrate the efficacy of CTCO in facilitating exploration in a real-world visual reaching task for a 7 DOF robotic arm with sparse rewards.

Details

ICML Conference 2022 Conference Paper

A Temporal-Difference Approach to Policy Gradient Estimation

Samuele Tosatto
Andrew Patterson
Martha White
A. Rupam Mahmood

The policy gradient theorem (Sutton et al. , 2000) prescribes the usage of a cumulative discounted state distribution under the target policy to approximate the gradient. Most algorithms based on this theorem, in practice, break this assumption, introducing a distribution shift that can cause the convergence to poor solutions. In this paper, we propose a new approach of reconstructing the policy gradient from the start state without requiring a particular sampling strategy. The policy gradient calculation in this form can be simplified in terms of a gradient critic, which can be recursively estimated due to a new Bellman equation of gradients. By using temporal-difference updates of the gradient critic from an off-policy data stream, we develop the first estimator that side-steps the distribution shift issue in a model-free way. We prove that, under certain realizability conditions, our estimator is unbiased regardless of the sampling strategy. We empirically show that our technique achieves a superior bias-variance trade-off and performance in presence of off-policy samples.

Details

ICRA Conference 2021 Conference Paper

Contextual Latent-Movements Off-Policy Optimization for Robotic Manipulation Skills

Samuele Tosatto
Georgia Chalvatzaki
Jan Peters 0001

Parameterized movement primitives have been extensively used for imitation learning of robotic tasks. However, the high-dimensionality of the parameter space hinders the improvement of such primitives in the reinforcement learning (RL) setting, especially for learning with physical robots. In this paper we propose a novel view on handling the demonstrated trajectories for acquiring low-dimensional, non-linear latent dynamics, using mixtures of probabilistic principal component analyzers (MPPCA) on the movements’ parameter space. Moreover, we introduce a new contextual off-policy RL algorithm, named LAtent-Movements Policy Optimization (LAMPO). LAMPO can provide gradient estimates from previous experience using self-normalized importance sampling, hence, making full use of samples collected in previous learning iterations. These advantages combined provide a complete framework for sample-efficient off-policy optimization of movement primitives for robot learning of high-dimensional manipulation skills. Our experimental results conducted both in simulation and on a real robot show that LAMPO provides sample-efficient policies against common approaches in literature. Code available at https://github.com/SamuelePolimi/lampo.

Details

RLDM Conference 2019 Conference Abstract

Non-Parametric Off-Policy Policy Gradient

Samuele Tosatto

Policy gradients methods typically require a sample estimate of the state distribution induced by the policy, which results into excessive interaction with the environment after each policy update in order to avoid poor gradient estimation. In real applications, such as robotics, sample efficiency is a critical aspect. Off-Policy policy gradient algorithms has been yet proposed in the literature, but they classically require the introduction of some strong approximations. For this reason, many works which relies on the off-policy policy gradient theorem, need to ensure the behavioral policy to be close to the optimization policy, eluding de facto the benefits provided by being off-policy. We well define these source of approximations, and we propose an off-policy algorithm which does not rely on these approximations, leading to a better off-policy gradient estimation. We employ kernel regression and density estimation to obtain an approximation of both the value function and the state-distribution in closed form. This estimation directly yields an approximated analytical solution for the policy gradient. We show that the resulting policy gradient estimate is surprisingly accurate even with a fixed small amount of off-policy samples.

PDF Details

ICML Conference 2017 Conference Paper

Boosted Fitted Q-Iteration

Samuele Tosatto
Matteo Pirotta
Carlo D'Eramo
Marcello Restelli

This paper is about the study of B-FQI, an Approximated Value Iteration (AVI) algorithm that exploits a boosting procedure to estimate the action-value function in reinforcement learning problems. B-FQI is an iterative off-line algorithm that, given a dataset of transitions, builds an approximation of the optimal action-value function by summing the approximations of the Bellman residuals across all iterations. The advantage of such approach w. r. t. to other AVI methods is twofold: (1) while keeping the same function space at each iteration, B-FQI can represent more complex functions by considering an additive model; (2) since the Bellman residual decreases as the optimal value function is approached, regression problems become easier as iterations proceed. We study B-FQI both theoretically, providing also a finite-sample error upper bound for it, and empirically, by comparing its performance to the one of FQI in different domains and using different regression techniques.

Details