Arrow Research search

Author name cluster

Mahdi Kallel

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers
2 author rows

Possible papers

7

EWRL Workshop 2025 Workshop Paper

Revisiting Proximal Policy Optimization

  • Mahdi Kallel
  • Jose-Luis Holgado-Alvarez
  • Samuele Tosatto
  • Carlo D'Eramo

On-policy Reinforcement Learning (RL) offers desirable features such as stable learning, fewer policy updates, and the ability to evaluate a policy’s return during training. While recent efforts have focused on off-policy methods, achieving significant advancements, PPO remains the go-to algorithm for on-policy RL due to its apparent simplicity and effectiveness. Nonetheless, PPO is highly sensitive to hyperparameters and relies on subtle, often poorly documented adjustments that can critically affect its performance, thereby limiting its utility in complex scenarios. In this paper, we revisit the PPO algorithm by introducing principled enhancements that improve performance while eliminating the need for extensive hyperparameter tuning and implementation-specific optimizations. Our proposed approach, PPO+, is a principled adaptation of the PPO algorithm that strengthens its adherence to the on-policy objective, enhancing stability and efficiency. PPO+ sets a new state-of-the-art for deep on-policy RL on MuJoCo control problems while maintaining a straightforward implementation. PPO+ demonstrates significantly improved asymptotic performance over PPO, and a substantially reduced performance gap with off-policy algorithms in several challenging continuous control tasks. Beyond just performance, our findings offer a fresh perspective on on-policy RL.

PDF

EWRL Workshop 2024 Workshop Paper

Augmented Bayesian Policy Search

  • Mahdi Kallel
  • Debabrota Basu
  • Riad Akrour
  • Carlo D'Eramo

Deterministic policies are often preferred over stochastic ones when implemented on physical systems. They can prevent erratic and harmful behaviors while being easier to implement and interpret. However, in practice, exploration is largely performed by stochastic policies. First-order Bayesian Optimization (BO) methods offer a principled way of performing exploration using deterministic policies. This is done through a learned probabilistic model, in the form of a Gaussian Process (GP), of the objective function and its gradient. Nonetheless, such approaches treat policy search as a black-box problem, and thus, neglect the reinforcement learning nature of the problem. In this work, we leverage the performance difference lemma to introduce a novel mean function for the GP. This results in augmenting BO methods with the action-value function. Hence, we call our method Augmented Bayesian Search (ABS). Interestingly, this new mean function enhances the posterior gradient with the deterministic policy gradient, effectively bridging the gap between BO and policy gradient methods. The resulting algorithm combines the convenience of the direct policy search with the scalability of reinforcement learning. We validate ABS on high-dimensional locomotion problems and demonstrate competitive performance compared to existing direct policy search schemes.

PDF

ICLR Conference 2024 Conference Paper

Augmented Bayesian Policy Search

  • Mahdi Kallel
  • Debabrota Basu
  • Riad Akrour
  • Carlo D'Eramo

Deterministic policies are often preferred over stochastic ones when implemented on physical systems. They can prevent erratic and harmful behaviors while being easier to implement and interpret. However, in practice, exploration is largely performed by stochastic policies. First-order Bayesian Optimization (BO) methods offer a principled way of performing exploration using deterministic policies. This is done through a learned probabilistic model of the objective function and its gradient. Nonetheless, such approaches treat policy search as a black-box problem, and thus, neglect the reinforcement learning nature of the problem. In this work, we leverage the performance difference lemma to introduce a novel mean function for the probabilistic model. This results in augmenting BO methods with the action-value function. Hence, we call our method Augmented Bayesian Search (ABS). Interestingly, this new mean function enhances the posterior gradient with the deterministic policy gradient, effectively bridging the gap between BO and policy gradient methods. The resulting algorithm combines the convenience of the direct policy search with the scalability of reinforcement learning. We validate ABS on high-dimensional locomotion problems and demonstrate competitive performance compared to existing direct policy search schemes.

Details

RLC Conference 2024 Conference Paper

Cyclicity-Regularized Coordination Graphs

  • Oliver Järnefelt
  • Mahdi Kallel
  • Carlo D'Eramo

In parallel with the rise of the successful value function factorization approach, numerous recent studies on Cooperative Multi-Agent Reinforcement Learning (MARL) have explored the application of Coordination Graphs (CG) to model the communication requirements among the agent population. These coordination problems often exhibit structural sparsity, which facilitates accurate joint value function learning with CGs. Value-based methods necessitate the computation of argmaxes over the exponentially large joint action space, leading to the adoption of the max-sum method from the distributed constraint optimization (DCOP) literature. However, it has been empirically observed that the performance of max-sum deteriorates with an increase in the number of agents, attributed to the increased cyclicity of the graph. While previous works have tackled this issue by sparsifying the graph based on a metric of edge importance, thereby demonstrating improved performance, we argue that neglecting topological considerations during the sparsification procedure can adversely affect action selection. Consequently, we advocate for the explicit consideration of graph cyclicity alongside edge importances. We demonstrate that this approach results in superior performance across various challenging coordination problems.

PDF Details

EWRL Workshop 2024 Workshop Paper

Cyclicity-Regularized Coordination Graphs

  • Oliver Järnefelt
  • Mahdi Kallel
  • Carlo D'Eramo

In parallel with the rise of the successful value function factorization approach, numerous recent studies on Cooperative Multi-Agent Reinforcement Learning (MARL) have explored the application of Coordination Graphs (CG) to model the communication requirements among the agent population. These coordination problems often exhibit structural sparsity, which facilitates accurate joint value function learning with CGs. Value-based methods necessitate the computation of argmaxes over the exponentially large joint action space, leading to the adoption of the max-sum method from the distributed constraint optimization (DCOP) literature. However, it has been empirically observed that the performance of max-sum deteriorates with an increase in the number of agents, attributed to the increased cyclicity of the graph. While previous works have tackled this issue by sparsifying the graph based on a metric of edge importance, thereby demonstrating improved performance, we argue that neglecting topological considerations during the sparsification procedure can adversely affect action selection. Consequently, we advocate for the explicit consideration of graph cyclicity alongside edge importances. We demonstrate that this approach results in superior performance across various challenging coordination problems.

PDF

RLJ Journal 2024 Journal Article

Cyclicity-Regularized Coordination Graphs

  • Oliver Järnefelt
  • Mahdi Kallel
  • Carlo D'Eramo

In parallel with the rise of the successful value function factorization approach, numerous recent studies on Cooperative Multi-Agent Reinforcement Learning (MARL) have explored the application of Coordination Graphs (CG) to model the communication requirements among the agent population. These coordination problems often exhibit structural sparsity, which facilitates accurate joint value function learning with CGs. Value-based methods necessitate the computation of argmaxes over the exponentially large joint action space, leading to the adoption of the max-sum method from the distributed constraint optimization (DCOP) literature. However, it has been empirically observed that the performance of max-sum deteriorates with an increase in the number of agents, attributed to the increased cyclicity of the graph. While previous works have tackled this issue by sparsifying the graph based on a metric of edge importance, thereby demonstrating improved performance, we argue that neglecting topological considerations during the sparsification procedure can adversely affect action selection. Consequently, we advocate for the explicit consideration of graph cyclicity alongside edge importances. We demonstrate that this approach results in superior performance across various challenging coordination problems.

PDF Details

EWRL Workshop 2024 Workshop Paper

Revisiting On-Policy Deep Reinforcement Learning

  • Mahdi Kallel
  • Samuele Tosatto
  • Carlo D'Eramo

On-policy Reinforcement Learning (RL) offers several desirable properties, including more stable learning, less frequent policy changes, and the capacity to evaluate a policy's return during the learning process. Despite the considerable success of recent off-policy methods, their on-policy counterparts continue to lag in terms of asymptotic performance and sample efficiency. Proximal Policy Optimization (PPO) remains the de facto standard, despite its complexity and demonstrated sensitivity to hyperparameters. In this work, we introduce On-Policy Soft Actor-Critic (ON-SAC), a methodical adaptation of the Soft Actor-Critic (SAC) algorithm tailored for the on-policy setting. Our approach begins with the observation that the current on-policy algorithms do not use true on-policy gradients. We build on this observation to offer founded remedies for this problem. Our algorithm establishes a new state-of-the-art for deep on-policy RL, while simplifying the process by eliminating the need for trust-region methods and intricate critic learning schemes.

PDF
v2026.09.13