Author name cluster

Sandhya Saisubramanian

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

17 papers

2 author rows

ICRA Conference 2025 Conference Paper

Mitigating Side Effects in Multi-Agent Systems Using Blame Assignment

Pulkit Rustagi
Sandhya Saisubramanian

When independently trained or designed robots are deployed in a shared environment, their combined actions can lead to unintended negative side effects (NSEs). To ensure safe and efficient operation, robots must optimize task performance while minimizing the penalties associated with NSEs, balancing individual objectives with collective impact. We model the problem of mitigating NSEs in a cooperative multi-agent system as a bi-objective lexicographic decentralized Markov decision process. We assume independence of transitions and rewards with respect to the robots' tasks, but the joint NSE penalty creates a form of dependence in this setting. To improve scalability, the joint NSE penalty is decomposed into individual penalties for each robot using credit assignment, which facilitates decentralized policy computation. We empirically demonstrate, using mobile robots and in simulation, the effectiveness and scalability of our approach in mitigating NSEs. Code: https://tinyurl.com/RECON-NSE-Mitigation

Details

AAMAS Conference 2025 Conference Paper

Multi-Objective Planning with Contextual Lexicographic Reward Preferences

Pulkit Rustagi
Yashwanthi Anand
Sandhya Saisubramanian

Autonomous agents are often required to plan under multiple objectives whose preference ordering varies based on context. The agent may encounter multiple contexts during its course of operation, each imposing a distinct lexicographic ordering over the objectives, with potentially different reward functions associated with each context. Existing approaches to multi-objective planning typically consider a single preference ordering over the objectives, across the state space, and do not support planning under multiple objective orderings within an environment. We present Contextual Lexicographic Markov Decision Process (CLMDP), a framework that enables planning under varying lexicographic objective orderings, depending on the context. In a CLMDP, both the objective ordering at a state and the associated reward functions are determined by the context. We employ a Bayesian approach to infer a state-context mapping from expert trajectories. Our algorithm to solve a CLMDP first computes a policy for each objective ordering and then combines them into a single context-aware policy that is valid and cycle-free. The effectiveness of the proposed approach is evaluated in simulation and using a mobile robot.

PDF

RLC Conference 2025 Conference Paper

WOFOSTGym: A Crop Simulator for Learning Annual and Perennial Crop Management Strategies

William Solow
Sandhya Saisubramanian
Alan Fern

We introduce WOFOSTGym, a novel crop simulation environment designed to train reinforcement learning (RL) agents to optimize agromanagement decisions for annual and perennial crops in single and multi-farm settings. Effective crop management requires optimizing yield and economic returns while minimizing environmental impact, a complex sequential decision-making problem well suited for RL. However, the lack of simulators for perennial crops in multi-farm contexts has hindered RL applications in this domain. Existing crop simulators also do not support multiple annual crops. WOFOSTGym addresses these gaps by supporting 23 annual crops and two perennial crops, enabling RL agents to learn diverse agromanagement strategies in multi-year, multi-crop, and multi-farm settings. Our simulator offers a suite of challenging tasks for learning under partial observability, non-Markovian dynamics, and delayed feedback. WOFOSTGym's standard RL interface allows researchers without agricultural expertise to explore a wide range of agromanagement problems. Our experiments demonstrate the learned behaviors across various crop varieties and soil types, highlighting WOFOSTGym's potential for advancing RL-driven decision support in agriculture.

PDF Details

RLJ Journal 2025 Journal Article

WOFOSTGym: A Crop Simulator for Learning Annual and Perennial Crop Management Strategies

William Solow
Sandhya Saisubramanian
Alan Fern

PDF Details

AAMAS Conference 2024 Conference Paper

Minimizing Negative Side Effects in Cooperative Multi-Agent Systems using Distributed Coordination

Moumita Choudhury
Sandhya Saisubramanian
Hao Zhang
Shlomo Zilberstein

Autonomous agents in real-world environments may encounter undesirable outcomes or negative side effects (NSEs) when working collaboratively alongside other agents. We frame the challenge of minimizing NSEs in a multi-agent setting as a lexicographic decentralized Markov decision process in which we assume independence of rewards and transitions with respect to the primary assigned tasks, but allowing negative side effects to create a form of dependence among the agents. We present a lexicographic Q-learning approach to mitigate the NSEs using human feedback models while maintaining near-optimality with respect to the assigned tasks—up to some given slack. Our empirical evaluation across two domains demonstrates that our collaborative approach effectively mitigates NSEs, outperforming non-collaborative methods.

PDF

IJCAI Conference 2023 Conference Paper

Explanation-Guided Reward Alignment

Saaduddin Mahmud
Sandhya Saisubramanian
Shlomo Zilberstein

Agents often need to infer a reward function from observations to learn desired behaviors. However, agents may infer a reward function that does not align with the original intent because there can be multiple reward functions consistent with its observations. Operating based on such misaligned rewards can be risky. Furthermore, black-box representations make it difficult to verify the learned rewards and prevent harmful behavior. We present a framework for verifying and improving reward alignment using explanations and show how explanations can help detect misalignment and reveal failure cases in novel scenarios. The problem is formulated as inverse reinforcement learning from ranked trajectories. Verification tests created from the trajectory dataset are used to iteratively validate and improve reward alignment. The agent explains its learned reward and a tester signals whether the explanation passes the test. In cases where the explanation fails, the agent offers alternative explanations to gather feedback, which is then used to improve the learned reward. We analyze the efficiency of our approach in improving reward alignment using different types of explanations and demonstrate its effectiveness in five domains.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Planning and Learning for Non-markovian Negative Side Effects Using Finite State Controllers

Aishwarya Srivastava
Sandhya Saisubramanian
Praveen Paruchuri
Akshat Kumar
Shlomo Zilberstein

Autonomous systems are often deployed in the open world where it is hard to obtain complete specifications of objectives and constraints. Operating based on an incomplete model can produce negative side effects (NSEs), which affect the safety and reliability of the system. We focus on mitigating NSEs in environments modeled as Markov decision processes (MDPs). First, we learn a model of NSEs using observed data that contains state-action trajectories and severity of associated NSEs. Unlike previous works that associate NSEs with state-action pairs, our framework associates NSEs with entire trajectories, which is more general and captures non-Markovian dependence on states and actions. Second, we learn finite state controllers (FSCs) that predict NSE severity for a given trajectory and generalize well to unseen data. Finally, we develop a constrained MDP model that uses information from the underlying MDP and the learned FSC for planning while avoiding NSEs. Our empirical evaluation demonstrates the effectiveness of our approach in learning and mitigating Markovian and non-Markovian NSEs.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Planning and Learning for Reliable Autonomy in the Open World

Sandhya Saisubramanian

Safe and reliable decision-making is critical for long-term deployment of autonomous systems. Despite the recent advances in artificial intelligence, ensuring safe and reliable operation of human-aligned autonomous systems in open-world environments remains a challenge. My research focuses on developing planning and learning algorithms that support reliable autonomy in fully and partially observable environments, in the presence of uncertainty, limited information, and limited resources. This talk covers a summary of my recent research towards reliable autonomy.

PDF Details DOI

JAIR Journal 2022 Journal Article

Avoiding Negative Side Effects of Autonomous Systems in the Open World

Sandhya Saisubramanian
Ece Kamar
Shlomo Zilberstein

Autonomous systems that operate in the open world often use incomplete models of their environment. Model incompleteness is inevitable due to the practical limitations in precise model specification and data collection about open-world environments. Due to the limited fidelity of the model, agent actions may produce negative side effects (NSEs) when deployed. Negative side effects are undesirable, unmodeled effects of agent actions on the environment. NSEs are inherently challenging to identify at design time and may affect the reliability, usability and safety of the system. We present two complementary approaches to mitigate the NSE via: (1) learning from feedback, and (2) environment shaping. The solution approaches target settings with different assumptions and agent responsibilities. In learning from feedback, the agent learns a penalty function associated with a NSE. We investigate the efficiency of different feedback mechanisms, including human feedback and autonomous exploration. The problem is formulated as a multi-objective Markov decision process such that optimizing the agent’s assigned task is prioritized over mitigating NSE. A slack parameter denotes the maximum allowed deviation from the optimal expected reward for the agent’s task in order to mitigate NSE. In environment shaping, we examine how a human can assist an agent, beyond providing feedback, and utilize their broader scope of knowledge to mitigate the impacts of NSE. We formulate the problem as a human-agent collaboration with decoupled objectives. The agent optimizes its assigned task and may produce NSE during its operation. The human assists the agent by performing modest reconfigurations of the environment so as to mitigate the impacts of NSE, without affecting the agent’s ability to complete its assigned task. We present an algorithm for shaping and analyze its properties. Empirical evaluations demonstrate the trade-offs in the performance of different approaches in mitigating NSE in different settings.

PDF Details DOI

ICRA Conference 2022 Conference Paper

Metareasoning for Safe Decision Making in Autonomous Systems

Justin Svegliato
Connor Basich
Sandhya Saisubramanian
Shlomo Zilberstein

Although experts carefully specify the high-level decision-making models in autonomous systems, it is infeasible to guarantee safety across every scenario during operation. We therefore propose a safety metareasoning system that optimizes the severity of the system's safety concerns and the interference to the system's task: the system executes in parallel a task process that completes a specified task and safety processes that each address a specified safety concern with a conflict resolver for arbitration. This paper offers a formal definition of a safety metareasoning system, a recommendation algorithm for a safety process, an arbitration algorithm for a conflict resolver, an application of our approach to planetary rover exploration, and a demonstration that our approach is effective in simulation.

Details

AAMAS Conference 2021 Conference Paper

Mitigating Negative Side Effects via Environment Shaping

Sandhya Saisubramanian
Shlomo Zilberstein

Agents operating in the open world often produce negative side effects (NSE), which are difficult to identify at design time. We examine how a human can assist an agent, beyond providing feedback, and exploit their broader scope of knowledge to mitigate the impacts of NSE. We formulate this problem as a human-agent team with decoupled objectives. The agent optimizes its assigned task, during which its actions may produce NSE. The human shapes the environment through minor reconfiguration actions so as to mitigate the impacts of agent’s side effects, without significantly degrading agent performance. We present an algorithm to solve this problem. Empirical evaluation shows that the proposed framework can successfully mitigate NSE, without affecting the agent’s ability to complete its assigned task.

PDF

IJCAI Conference 2020 Conference Paper

A Multi-Objective Approach to Mitigate Negative Side Effects

Sandhya Saisubramanian
Ece Kamar
Shlomo Zilberstein

Agents operating in unstructured environments often create negative side effects (NSE) that may not be easy to identify at design time. We examine how various forms of human feedback or autonomous exploration can be used to learn a penalty function associated with NSE during system deployment. We formulate the problem of mitigating the impact of NSE as a multi-objective Markov decision process with lexicographic reward preferences and slack. The slack denotes the maximum deviation from an optimal policy with respect to the agent's primary objective allowed in order to mitigate NSE as a secondary objective. Empirical evaluation of our approach shows that the proposed framework can successfully mitigate NSE and that different feedback mechanisms introduce different biases, which influence the identification of NSE.

PDF Details DOI

AAAI Conference 2019 Short Paper

Adaptive Modeling for Risk-Aware Decision Making

Sandhya Saisubramanian

This thesis aims to provide a foundation for risk-aware decision making. Decision making under uncertainty is a core capability of an autonomous agent. A cornerstone for with long-term autonomy and safety is risk-aware decision making. A risk-aware model fully accounts for a known set of risks in the environment, with respect to the problem under consideration, and the process of decision making using such a model is risk-aware decision making. Formulating risk-aware models is critical for robust reasoning under uncertainty, since the impact of using less accurate models may be catastrophic in extreme cases due to overly optimistic view of problems. I propose adaptive modeling, a framework that helps balance the trade-off between model simplicity and risk awareness, for different notions of risks, while remaining computationally tractable.

PDF Details

IROS Conference 2019 Conference Paper

Adaptive Outcome Selection for Planning with Reduced Models

Sandhya Saisubramanian
Shlomo Zilberstein

Reduced models allow autonomous robots to cope with the complexity of planning in stochastic environments by simplifying the model and reducing its accuracy. The solution quality of a reduced model depends on its fidelity. We present 0/1 reduced model that selectively improves model fidelity in certain states by switching between using a simplified deterministic model and the full model, without significantly compromising the run time gains. We measure the reduction impact for a reduced model based on the values of the ignored outcomes and use this as a heuristic for outcome selection. Finally, we present empirical results of our approach on three different domains, including an electric vehicle charging problem using real-world data from a university campus.

Details

IROS Conference 2019 Conference Paper

Planning in Stochastic Environments with Goal Uncertainty

Sandhya Saisubramanian
Kyle Hollins Wray
Luis Enrique Pineda
Shlomo Zilberstein

We present the Goal Uncertain Stochastic Shortest Path (GUSSP) problem — a general framework to model path planning and decision making in stochastic environments with goal uncertainty. The framework extends the stochastic shortest path (SSP) model to dynamic environments in which it is impossible to determine the exact goal states ahead of plan execution. GUSSPs introduce flexibility in goal specification by allowing a belief over possible goal configurations. The unique observations at potential goals helps the agent identify the true goal during plan execution. The partial observability is restricted to goals, facilitating the reduction to an SSP with a modified state space. We formally define a GUSSP and discuss its theoretical properties. We then propose an admissible heuristic that reduces the planning time using FLARES — a start-of-the-art probabilistic planner. We also propose a determinization approach for solving this class of problems. Finally, we present empirical results on a search and rescue mobile robot and three other problem domains in simulation.

Details

AAMAS Conference 2018 Conference Paper

Planning Using a Portfolio of Reduced Models

Sandhya Saisubramanian
Shlomo Zilberstein
Prashant Shenoy

Existing reduced model techniques simplify a problem by applying a uniform principle to reduce the number of considered outcomes for all state-action pairs. It is non-trivial to identify which outcome selection principle will work well across all problem instances in a domain. We aim to create reduced models that yield near-optimal solutions, without compromising the run time gains of using a reduced model. First, we introduce planning using a portfolio of reduced models, a framework that provides flexibility in the reduced model formulation by using a portfolio of outcome selection principles. Second, we propose planning using cost adjustment, a technique that improves the solution quality by accounting for the outcomes ignored in the reduced model. Empirical evaluation of these techniques confirm their effectiveness in several domains.

PDF

AAAI Conference 2015 Conference Paper

Risk Based Optimization for Improving Emergency Medical Systems

Sandhya Saisubramanian
Pradeep Varakantham
Hoong Chuin Lau

In emergency medical systems, arriving at the incident location a few seconds early can save a human life. Thus, this paper is motivated by the need to reduce the response time – time taken to arrive at the incident location after receiving the emergency call – of Emergency Response Vehicles, ERVs (ex: ambulances, ﬁre rescue vehicles) for as many requests as possible. We expect to achieve this primarily by positioning the ”right” number of ERVs at the ”right” places and at the ”right” times. Given the exponentially large action space (with respect to number of ERVs and their placement) and the stochasticity in location and timing of emergency incidents, this problem is computationally challenging. To that end, our contributions building on existing data-driven approaches are three fold: 1. Based on real world evaluation metrics, we provide a risk based optimization criterion to learn from past incident data. Instead of minimizing expected response time, we minimize the largest value of response time such that the risk of ﬁnding requests that have a higher value is bounded (ex: Only 10% of requests should have a response time greater than 8 minutes). 2. We develop a mixed integer linear optimization formulation to learn and compute an allocation from a set of input requests while considering the risk criterion. 3. To allow for ”live” reallocation of ambulances, we provide a decomposition method based on Lagrangian Relaxation to signiﬁcantly reduce the run-time of the optimization formulation. Finally, we provide an exhaustive evaluation on real-world datasets from two asian cities that demonstrates the improvement provided by our approach over current practice and the best known approach from literature.

PDF Details