Author name cluster

Mehdi Fatemi

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

14 papers

2 author rows

ICLR Conference 2024 Conference Paper

A Dynamical View of the Question of Why

Mehdi Fatemi
Sindhu C. M. Gowda

We address causal reasoning in multivariate time series data generated by stochastic processes. Existing approaches are largely restricted to static settings, ignoring the continuity and emission of variations across time. In contrast, we propose a learning paradigm that directly establishes causation between events in the course of time. We present two key lemmas to compute causal contributions and frame them as reinforcement learning problems. Our approach offers formal and computational tools for uncovering and quantifying causal relationships in diffusion processes, subsuming various important settings such as discrete-time Markov decision processes. Finally, in fairly intricate experiments and through sheer learning, our framework reveals and quantifies causal links, which otherwise seem inexplicable.

Details

NeurIPS Conference 2024 Conference Paper

OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset

Allen Roush
Yusuf Shabazz
Arvind Balaji
Peter Zhang
Stefano Mezza
Markus Zhang
Sanjay Basu
Sriram Vishwanath

We introduce OpenDebateEvidence, a comprehensive dataset for argument miningand summarization sourced from the American Competitive Debate community. This dataset includes over 3. 5 million documents with rich metadata, making itone of the most extensive collections of debate evidence. OpenDebateEvidencecaptures the complexity of arguments in high school and college debates, pro-viding valuable resources for training and evaluation. Our extensive experimentsdemonstrate the efficacy of fine-tuning state-of-the-art large language models forargumentative abstractive summarization across various methods, models, anddatasets. By providing this comprehensive resource, we aim to advance com-putational argumentation and support practical applications for debaters, edu-cators, and researchers. OpenDebateEvidence is publicly available to supportfurther research and innovation in computational argumentation. Access it here: https: //huggingface. co/datasets/Yusuf5/OpenCaselist.

PDF Details DOI

ICML Conference 2024 Conference Paper

Successor Features for Efficient Multi-Subject Controlled Text Generation

Meng Cao 0003
Mehdi Fatemi
Jackie C. K. Cheung
Samira Shabanian

While large language models (LLMs) have achieved impressive performance in generating fluent and realistic text, controlling the generated text so that it exhibits properties such as safety, factuality, and non-toxicity remains challenging. Existing decoding-based controllable text generation methods are static in terms of the dimension of control; if the target subject is changed, they require new training. Moreover, it can quickly become prohibitive to concurrently control multiple subjects. To address these challenges, we first show that existing methods can be framed as a reinforcement learning problem, where an action-value function estimates the likelihood of a desired attribute appearing in the generated text. Then, we introduce a novel approach named SF-Gen, which leverages the concept of successor features to decouple the dynamics of LLMs from task-specific rewards. By employing successor features, our method proves to be memory-efficient and computationally efficient for both training and decoding, especially when dealing with multiple target subjects. To the best of our knowledge, our research represents the first application of successor features in text generation. In addition to its computational efficiency, the resultant language produced by our method is comparable to the SOTA (and outperforms baselines) in both control measures as well as language quality, which we demonstrate through a series of experiments in various controllable text generation tasks.

Details

ICLR Conference 2023 Conference Paper

Systematic Rectification of Language Models via Dead-end Analysis

Meng Cao 0003
Mehdi Fatemi
Jackie C. K. Cheung
Samira Shabanian

With adversarial or otherwise normal prompts, existing large language models (LLM) can be pushed to generate toxic discourses. One way to reduce the risk of LLMs generating undesired discourses is to alter the training of the LLM. This can be very restrictive due to demanding computation requirements. Other methods rely on rule-based or prompt-based token elimination, which are limited as they dismiss future tokens and the overall meaning of the complete discourse. Here, we center detoxification on the probability that the finished discourse is ultimately considered toxic. That is, at each point, we advise against token selections proportional to how likely a finished text from this point will be toxic. To this end, we formally extend the dead-end theory from the recent reinforcement learning (RL) literature to also cover uncertain outcomes. Our approach, called rectification, utilizes a separate but significantly smaller model for detoxification, which can be applied to diverse LLMs as long as they share the same vocabulary. Importantly, our method does not require access to the internal representations of the LLM, but only the token probability distribution at each decoding step. We believe this is important since many LLMs today are hosted in servers and only accessible through APIs. When applied to various LLMs, including GPT-3, our approach generates notably better results compared to the base LLMs and other techniques in terms of the overall language and detoxification performance.

Details

ICLR Conference 2022 Conference Paper

Orchestrated Value Mapping for Reinforcement Learning

Mehdi Fatemi
Arash Tavakoli

We present a general convergent class of reinforcement learning algorithms that is founded on two distinct principles: (1) mapping value estimates to a different space using arbitrary functions from a broad class, and (2) linearly decomposing the reward signal into multiple channels. The first principle enables incorporating specific properties into the value estimator that can enhance learning. The second principle, on the other hand, allows for the value function to be represented as a composition of multiple utility functions. This can be leveraged for various purposes, e.g. dealing with highly varying reward scales, incorporating a priori knowledge about the sources of reward, and ensemble learning. Combining the two principles yields a general blueprint for instantiating convergent algorithms by orchestrating diverse mapping functions over multiple reward channels. This blueprint generalizes and subsumes algorithms such as Q-Learning, Log Q-Learning, and Q-Decomposition. In addition, our convergence proof for this general class relaxes certain required assumptions in some of these algorithms. Based on our theory, we discuss several interesting configurations as special cases. Finally, to illustrate the potential of the design space that our theory opens up, we instantiate a particular algorithm and evaluate its performance on the Atari suite.

Details

ICLR Conference 2021 Conference Paper

Learning to Represent Action Values as a Hypergraph on the Action Vertices

Arash Tavakoli
Mehdi Fatemi
Petar Kormushev

Action-value estimation is a critical component of many reinforcement learning (RL) methods whereby sample complexity relies heavily on how fast a good estimator for action value can be learned. By viewing this problem through the lens of representation learning, good representations of both state and action can facilitate action-value estimation. While advances in deep learning have seamlessly driven progress in learning state representations, given the specificity of the notion of agency to RL, little attention has been paid to learning action representations. We conjecture that leveraging the combinatorial structure of multi-dimensional action spaces is a key ingredient for learning good representations of action. To test this, we set forth the action hypergraph networks framework---a class of functions for learning action representations in multi-dimensional discrete action spaces with a structural inductive bias. Using this framework we realise an agent class based on a combination with deep Q-networks, which we dub hypergraph Q-networks. We show the effectiveness of our approach on a myriad of domains: illustrative prediction problems under minimal confounding effects, Atari 2600 games, and discretised physical control benchmarks.

Details

NeurIPS Conference 2021 Conference Paper

Medical Dead-ends and Learning to Identify High-Risk States and Treatments

Mehdi Fatemi
Taylor W. Killian
Jayakumar Subramanian
Marzyeh Ghassemi

Machine learning has successfully framed many sequential decision making problems as either supervised prediction, or optimal decision-making policy identification via reinforcement learning. In data-constrained offline settings, both approaches may fail as they assume fully optimal behavior or rely on exploring alternatives that may not exist. We introduce an inherently different approach that identifies "dead-ends" of a state space. We focus on patient condition in the intensive care unit, where a "medical dead-end" indicates that a patient will expire, regardless of all potential future treatment sequences. We postulate "treatment security" as avoiding treatments with probability proportional to their chance of leading to dead-ends, present a formal proof, and frame discovery as an RL problem. We then train three independent deep neural models for automated state construction, dead-end discovery and confirmation. Our empirical results discover that dead-ends exist in real clinical data among septic patients, and further reveal gaps between secure treatments and those administered.

PDF Details

ICML Conference 2021 Conference Paper

Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks

Sungryull Sohn
Sungtae Lee
Jongwook Choi
Harm van Seijen
Mehdi Fatemi
Honglak Lee

We propose the k-Shortest-Path (k-SP) constraint: a novel constraint on the agent’s trajectory that improves the sample efficiency in sparse-reward MDPs. We show that any optimal policy necessarily satisfies the k-SP constraint. Notably, the k-SP constraint prevents the policy from exploring state-action pairs along the non-k-SP trajectories (e. g. , going back and forth). However, in practice, excluding state-action pairs may hinder the convergence of RL algorithms. To overcome this, we propose a novel cost function that penalizes the policy violating SP constraint, instead of completely excluding it. Our numerical experiment in a tabular RL setting demonstrates that the SP-constraint can significantly reduce the trajectory space of policy. As a result, our constraint enables more sample efficient learning by suppressing redundant exploration and exploitation. Our experiments on MiniGrid, DeepMind Lab, Atari, and Fetch show that the proposed method significantly improves proximal policy optimization (PPO) and outperforms existing novelty-seeking exploration methods including count-based exploration even in continuous control tasks, indicating that it improves the sample efficiency by preventing the agent from taking redundant actions.

Details

RLDM Conference 2019 Conference Abstract

Dead-ends and Secure Exploration in Reinforcement Learning

Mehdi Fatemi
Shikhar Sharma
Samira Ebrahimi Kahou

Many interesting applications of reinforcement learning (RL) involve MDPs that include many “dead-end” states. Upon reaching a dead-end state, the agent continues to interact with the environment in a dead-end trajectory before reaching a terminal state, but cannot collect any positive reward, regardless of whatever actions are chosen by the agent. The situation is even worse when existence of many dead- end states is coupled with distant positive rewards from any initial state (we call it Bridge Effect). Hence, conventional exploration techniques often incur prohibitively large training steps before convergence. To deal with the bridge effect, we propose a condition for exploration, called security. We next establish formal results that translate the security condition into the learning problem of an auxiliary value function. This new value function is used to cap “any” given exploration policy and is guaranteed to make it secure. As a special case, we use this theory and introduce secure random-walk. We next extend our results to the deep RL settings by identifying and addressing two main challenges that arise. Finally, we empirically compare secure random-walk with standard benchmarks in two sets of experiments including the Atari game of Montezuma’s Revenge.

PDF Details

ICML Conference 2019 Conference Paper

Dead-ends and Secure Exploration in Reinforcement Learning

Mehdi Fatemi
Shikhar Sharma 0001
Harm van Seijen
Samira Ebrahimi Kahou

Many interesting applications of reinforcement learning (RL) involve MDPs that include numerous “dead-end" states. Upon reaching a dead-end state, the agent continues to interact with the environment in a dead-end trajectory before reaching an undesired terminal state, regardless of whatever actions are chosen. The situation is even worse when existence of many dead-end states is coupled with distant positive rewards from any initial state (we term this as Bridge Effect). Hence, conventional exploration techniques often incur prohibitively many training steps before convergence. To deal with the bridge effect, we propose a condition for exploration, called security. We next establish formal results that translate the security condition into the learning problem of an auxiliary value function. This new value function is used to cap “any" given exploration policy and is guaranteed to make it secure. As a special case, we use this theory and introduce secure random-walk. We next extend our results to the deep RL settings by identifying and addressing two main challenges that arise. Finally, we empirically compare secure random-walk with standard benchmarks in two sets of experiments including the Atari game of Montezuma’s Revenge.

Details

RLDM Conference 2019 Conference Abstract

Using a Logarithmic Mapping to Enable Lower Discount Factors

Harm H van Seijen
Mehdi Fatemi
Kimia Nadjahi

The discount factor in reinforcement learning is often used as a hyperparameter. It affects learning properties such as speed of learning, generalization behaviour, and sensitivity to function approximation errors in a significant way. Its optimal value is the result of a trade-off between the different ways it affects the optimization process. We identify a phenomenon that negatively impacts the performance of methods that combine function approximation with a discounted objective, especially when relatively small discount factors are used: the action gap can vary orders of magnitude in size across the state space. Therefore, it can be exponentially harder for SGD-based methods to improve the policy in one area of the state-space compared to another area. To achieve a more homogeneous action-gap across the state space, we propose to map the value function to a logarithmic space and perform the SGD-updates in this space instead. For deterministic environments, we prove that this technique results in convergence to the optimal policy under standard assumptions. Empirically, we demonstrate that it can yield large performance improvements, by evaluating it on a large set of (stochastic) Atari games.

PDF Details

NeurIPS Conference 2019 Conference Paper

Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning

Harm van Seijen
Mehdi Fatemi
Arash Tavakoli

In an effort to better understand the different ways in which the discount factor affects the optimization process in reinforcement learning, we designed a set of experiments to study each effect in isolation. Our analysis reveals that the common perception that poor performance of low discount factors is caused by (too) small action-gaps requires revision. We propose an alternative hypothesis that identifies the size-difference of the action-gap across the state-space as the primary cause. We then introduce a new method that enables more homogeneous action-gaps by mapping value estimates to a logarithmic space. We prove convergence for this method under standard assumptions and demonstrate empirically that it indeed enables lower discount factors for approximate reinforcement-learning methods. This in turn allows tackling a class of reinforcement-learning problems that are challenging to solve with traditional methods.

PDF Details

NeurIPS Conference 2017 Conference Paper

Hybrid Reward Architecture for Reinforcement Learning

Harm van Seijen
Mehdi Fatemi
Joshua Romoff
Romain Laroche
Tavian Barnes
Jeffrey Tsang

One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.

PDF Details

RLDM Conference 2017 Conference Abstract

Separation of Concerns in Reinforcement Learning

Harm van Seijen
Mehdi Fatemi
Joshua Romoff

In this paper, we propose a model for solving a single-agent task by using multiple agents, each focusing on a different aspect of the task. This approach has two main advantages: 1) it allows for training specialized agents on different parts of the task, and 2) it provides a new way to transfer knowledge, by transferring trained agents. Our model generalizes the traditional hierarchical decomposition, where agents are organized in a hierarchical way. We validate our model empirically, by showing that it can beat a single- agent state-of-the-art method on a challenging domain.

PDF Details