Author name cluster

Sushant Veer

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

16 papers

2 author rows

ICRA Conference 2025 Conference Paper

LoRD: Adapting Differentiable Driving Policies to Distribution Shifts

Christopher Diehl
Péter Karkus
Sushant Veer
Marco Pavone 0001
Torsten Bertram

Distribution shifts between operational domains can severely affect the performance of learned models in self-driving vehicles (SDVs). While this is a well-established problem, prior work has mostly explored naive solutions such as fine-tuning, focusing on the motion prediction task. In this work, we explore novel adaptation strategies for differentiable autonomy stacks (structured policy) consisting of prediction, planning, and control, perform evaluation in closed-loop, and investigate the often-overlooked issue of catastrophic forgetting. Specifically, we introduce two simple yet effective techniques: a low-rank residual decoder (LoRD) and multi-task fine-tuning. Through experiments across three models conducted on two real-world autonomous driving datasets (nuPlan, exiD), we demonstrate the effectiveness of our methods and highlight a significant performance gap between open-loop and closed-loop evaluation in prior approaches. Our approach improves forgetting by up to 23. 33% and the closed-loop out-of-distribution driving score by 9. 93% in comparison to standard fine-tuning. https://github.com/rst-tu-dortmund/LoRD

Details

ICRA Conference 2025 Conference Paper

Online Aggregation of Trajectory Predictors

Alex Tong
Apoorva Sharma
Sushant Veer
Marco Pavone 0001
Heng Yang

Trajectory prediction, the task of forecasting future agent behavior from past data, is central to safe and efficient autonomous driving. A diverse set of methods (e. g. , rule-based or learned with different architectures and datasets) have been proposed, yet it is often the case that the performance of these methods is sensitive to the deployment environment (e. g. , how well the design rules model the environment, or how accurately the test data match the training data). Building upon the principled theory of online convex optimization but also going beyond convexity and stationarity, we present a lightweight and model-agnostic method to aggregate different trajectory predictors online. We propose treating each individual trajectory predictor as an “expert” and maintaining a probability vector to mix the outputs of different experts. Then, the key technical approach lies in leveraging online data - the true agent behavior to be revealed at the next timestepto form a convex-or-nonconvex, stationary-or-dynamic loss function whose gradient steers the probability vector towards choosing the best mixture of experts. We instantiate this method to aggregate trajectory predictors trained on different cities in the nuScenes dataset and show that it performs just as well, if not better than, any singular model, even when deployed on the out-of-distribution LYFT dataset.

Details

ICRA Conference 2025 Conference Paper

System-Level Safety Monitoring and Recovery for Perception Failures in Autonomous Vehicles

Kaustav Chakraborty
Zeyuan Feng
Sushant Veer
Apoorva Sharma
Boris Ivanovic
Marco Pavone 0001
Somil Bansal

The safety-critical nature of autonomous vehicle (AV) operation necessitates development of task-relevant algorithms that can reason about safety at the system level and not just at the component level. To reason about the impact of a perception failure on the entire system performance, such task-relevant algorithms must contend with various challenges: complexity of AV stacks, high uncertainty in the operating environments, and the need for real-time performance. To overcome these challenges, in this work, we introduce a Q-network called SPARQ (abbreviation for Safety evaluation for Perception And Recovery Q-network) that evaluates the safety of a plan generated by a planning algorithm, accounting for perception failures that the planning process may have over-looked. This Q-network can be queried during system runtime to assess whether a proposed plan is safe for execution or poses potential safety risks. If a violation is detected, the network can then recommend a corrective plan while accounting for the perceptual failure. We validate our algorithm using the NuPlan-Vegas dataset, demonstrating its ability to handle cases where a perception failure compromises a proposed plan while the corrective plan remains safe. We observe an overall accuracy and recall of 90% while sustaining a frequency of 42Hz on the unseen testing dataset. We compare our performance to a popular reachability - based baseline and analyze some interesting properties of our approach in improving the safety properties of an AV pipeline. Website: vatsuak.github.io/sparq

Details

TMLR Journal 2025 Journal Article

Wolf: Dense Video Captioning with a World Summarization Framework

Boyi Li
Ligeng Zhu
Ran Tian
Shuhan Tan
Yuxiao Chen
Yao Lu
Yin Cui
Sushant Veer

We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision Language Models (VLMs). By utilizing both image and video models, our framework captures different levels of information and summarizes them efficiently. Our approach can be applied to enhance video understanding, auto-labeling, and captioning. To evaluate caption quality, we introduce CapScore, an LLM-based metric to assess the similarity and quality of generated captions compared to the ground truth captions. We further build four human-annotated datasets in three domains: autonomous driving, general scenes, and robotics, to facilitate comprehensive comparisons. We show that Wolf achieves superior captioning performance compared to state-of-the-art approaches from the research community (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For instance, in comparison with GPT-4V, Wolf improves CapScore (caption quality) by 55.6% and CapScore (caption similarity) by 77.4% on challenging driving videos. Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment.

PDF Details

ICRA Conference 2023 Conference Paper

Guided Conditional Diffusion for Controllable Traffic Simulation

Ziyuan Zhong
Davis Rempe
Danfei Xu
Yuxiao Chen 0008
Sushant Veer
Tong Che
Baishakhi Ray
Marco Pavone 0001

Controllable and realistic traffic simulation is critical for developing and verifying autonomous vehicles. Typical heuristic-based traffic models offer flexible control to make vehicles follow specific trajectories and traffic rules. On the other hand, data-driven approaches generate realistic and human-like behaviors, improving transfer from simulated to real-world traffic. However, to the best of our knowledge, no traffic model offers both controllability and realism. In this work, we develop a conditional diffusion model for controllable traffic generation (CTG) that allows users to control desired properties of trajectories at test time (e. g. , reach a goal or follow a speed limit) while maintaining realism and physical feasibility through enforced dynamics. The key technical idea is to leverage recent advances from diffusion modeling and differentiable logic to guide generated trajectories to meet rules defined using signal temporal logic (STL). We further extend guidance to multi-agent settings and enable interaction-based rules like collision avoidance. CTG is extensively evaluated on the nuScenes dataset for diverse and composite rules, demonstrating improvement over strong baselines in terms of the controllability-realism tradeoff. Demo videos can be found at https://aiasd.github.io/ctg.github.io

Details

NeurIPS Conference 2023 Conference Paper

PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction

Apoorva Sharma
Sushant Veer
Asher Hancock
Heng Yang
Marco Pavone
Anirudha Majumdar

Inductive Conformal Prediction (ICP) provides a practical and effective approach for equipping deep learning models with uncertainty estimates in the form of set-valued predictions which are guaranteed to contain the ground truth with high probability. Despite the appeal of this coverage guarantee, these sets may not be efficient: the size and contents of the prediction sets are not directly controlled, and instead depend on the underlying model and choice of score function. To remedy this, recent work has proposed learning model and score function parameters using data to directly optimize the efficiency of the ICP prediction sets. While appealing, the generalization theory for such an approach is lacking: direct optimization of empirical efficiency may yield prediction sets that are either no longer efficient on test data, or no longer obtain the required coverage on test data. In this work, we use PAC-Bayes theory to obtain generalization bounds on both the coverage and the efficiency of set-valued predictors which can be directly optimized to maximize efficiency while satisfying a desired test coverage. In contrast to prior work, our framework allows us to utilize the entire calibration dataset to learn the parameters of the model and score function, instead of requiring a separate hold-out set for obtaining test-time coverage guarantees. We leverage these theoretical results to provide a practical algorithm for using calibration data to simultaneously fine-tune the parameters of a model and score function while guaranteeing test-time coverage and efficiency of the resulting prediction sets. We evaluate the approach on regression and classification tasks, and outperform baselines calibrated using a Hoeffding bound-based PAC guarantee on ICP, especially in the low-data regime.

PDF Details

ICRA Conference 2023 Conference Paper

Receding Horizon Planning with Rule Hierarchies for Autonomous Vehicles

Sushant Veer
Karen Leung
Ryan K. Cosner
Yuxiao Chen 0008
Péter Karkus
Marco Pavone 0001

Autonomous vehicles must often contend with conflicting planning requirements, e. g. , safety and comfort could be at odds with each other if avoiding a collision calls for slamming the brakes. To resolve such conflicts, assigning importance ranking to rules (i. e. , imposing a rule hierarchy) has been proposed, which, in turn, induces rankings on trajectories based on the importance of the rules they satisfy. On one hand, imposing rule hierarchies can enhance interpretability, but introduce combinatorial complexity to planning; while on the other hand, differentiable reward structures can be leveraged by modern gradient-based optimization tools, but are less interpretable and unintuitive to tune. In this paper, we present an approach to equivalently express rule hierar-chies as differentiable reward structures amenable to modern gradient-based optimizers, thereby, achieving the best of both worlds. We achieve this by formulating rank-preserving reward functions that are monotonic in the rank of the trajectories induced by the rule hierarchy; i. e. , higher ranked trajectories receive higher reward. Equipped with a rule hierarchy and its corresponding rank-preserving reward function, we develop a two-stage planner that can efficiently resolve conflicting planning requirements. We demonstrate that our approach can generate motion plans in ~7-10 Hz for various challenging road navigation and intersection negotiation scenarios.

Details

ICRA Conference 2022 Conference Paper

Stronger Generalization Guarantees for Robot Learning by Combining Generative Models and Real-World Data

Abhinav Agarwal
Sushant Veer
Allen Z. Ren
Anirudha Majumdar

We are motivated by the problem of learning policies for robotic systems with rich sensory inputs (e. g. , vision) in a manner that allows us to guarantee generalization to environments unseen during training. We provide a framework for providing such generalization guarantees by leveraging a finite dataset of real-world environments in combination with a (potentially inaccurate) generative model of environments. The key idea behind our approach is to utilize the generative model in order to implicitly specify a prior over policies. This prior is updated using the real-world dataset of environments by minimizing an upper bound on the expected cost across novel environments derived via Probably Approximately Correct (PAC)-Bayes generalization theory. We demonstrate our approach on two simulated systems with nonlinear/hybrid dynamics and rich sensing modalities: (i) quadrotor navigation with an onboard vision sensor, and (ii) grasping objects using a depth sensor. Comparisons with prior work demonstrate the ability of our approach to obtain stronger generalization guarantees by utilizing generative models. We also present hardware experiments for validating our bounds for the grasping task.

Details

ICRA Conference 2020 Conference Paper

An Adaptive Supervisory Control Approach to Dynamic Locomotion Under Parametric Uncertainty

Prem Chand
Sushant Veer
Ioannis Poulakakis

This paper presents an adaptive control scheme for robotic systems that operate in the face of-potentially large-structured uncertainty. The proposed adaptive controller employs an on-line supervisor that utilizes logic-based switching among a finite set of controllers to identify uncertain parameters, and adapt the behavior of the system based on a current estimate of their value. To achieve this, the adaptive control approach in this paper combines on-line parameter estimation and feedback control while avoiding some of the inherent difficulties of classical adaptive control strategies. Furthermore, the proposed supervisory control architecture is modular as it relies on established "off-the-shelf" feedback control law and estimator design approaches, instead of cus-tomizing the overall design to the specific requirements of an adaptive control algorithm. We demonstrate the efficacy of the method on the problem of a dynamically-walking bipedal robot delivering a payload of unknown mass, and show that, by switching to the controller that is the "best" according to a current estimate of the uncertainty, the system maintains a low energy cost during its operation.

Details

IROS Conference 2019 Conference Paper

PD based Robust Quadratic Programs for Robotic Systems

Shishir Kolathaya
Sushant Veer

In this paper, inspired by Proportional-Derivative (PD) control laws, we present a class of Control Lyapunov Function (CLF) based Quadratic Programs (QPs) for robotic systems. Proportional-Derivative (PD) control laws are independent of the robot model, however, they fail to incorporate physical constraints, such as torque saturation. On the other hand, most optimization based control design approaches ensure satisfaction of the physical constraints, but they are sensitive to errors in the robot model. The PD based Quadratic Programs (PD-QPs), presented in this paper, are a first step towards bridging this gap between the PD and the optimization based controllers to bring the best of both together. We derive two versions of PD-QPs: model-based and model-free. Furthermore, for tracking time-varying trajectories, we establish asymptotic stability for the model-based PD-QP, and ultimate boundedness for the model-free PD-QP. The performance of the PD-QPs is evaluated on two robot models: a fully actuated cart-pole and an underactuated 5-DOF biped.

Details

ICRA Conference 2019 Conference Paper

Safe Adaptive Switching among Dynamical Movement Primitives: Application to 3D Limit-Cycle Walkers

Sushant Veer
Ioannis Poulakakis

Complex robot motions are frequently generated by composing simpler primitive movements. We use this approach to formulate robot motion plans as sequences of primitives to be executed one after the other. When dealing with dynamical movement primitives, besides accomplishing the high-level objective, planners must also reason about the effect of the plan's execution on the safety of the platform. This task is exacerbated by the presence of disturbances, such as non-vanishing external forces. To address this issue, we present a framework that builds on rigorous control-theoretic tools to generate safely executable motion plans for externally excited robotic systems. We illustrate the proposed framework on adapting the motion of a 3D bipedal robot model to persistent external forcing by switching among dynamic movement primitives, each corresponding to a limit-cycle walking gait.

Details

IROS Conference 2017 Conference Paper

Adaptation of limit-cycle walkers for collaborative tasks: A supervisory switching control approach

Sushant Veer
Mohamad Shafiee Motahar
Ioannis Poulakakis

This paper presents a method to achieve online gait adaptation of a dynamically walking biped when collaborating with an external agent-either a human or a robot-acting as a leader. Adaptation occurs without any explicit information on the leader's intended motion; only implicit information is used through the interaction force developed between the leader and the biped. An adaptive supervisory control scheme is proposed and guarantees for boundedness of the state despite switching under external force are provided. The supervisory controller leverages the availability of a library of exponentially stable limit-cycle gaits, and orchestrates switching among them in an online fashion to achieve adaptation. As a result, the range of leader speeds that the biped can adapt to is drastically enlarged while the leader's effort is reduced.

Details

IROS Conference 2017 Conference Paper

Almost driftless navigation of 3D limit-cycle walking bipeds

Sushant Veer
Mohamad Shafiee Motahar
Ioannis Poulakakis

This paper presents a method for navigating 3D dynamically walking bipedal robots amidst obstacles. Our framework relies on composing gait primitives corresponding to limit-cycle locomotion behaviors and it produces nominal motion plans that are compatible with the system's dynamics and can be tracked with high fidelity. The low-level controllers of the biped are designed within the Hybrid Zero Dynamics (HZD) framework. Exploiting the dimensional reduction afforded by HZD and properties of invariant sets of switching systems among multiple equilibria, we obtain polynomial approximations of a reduced order Poincaré map and of the net change of the center of mass location over a stride. These polynomials are then incorporated in a high-level Rapidly Exploring Random Tree (RRT) planner to generate nominal plans which are tracked by the biped with drastically low drifting errors, without adversely affecting the time for computation.

Details

IROS Conference 2017 Conference Paper

Steering a 3D limit-cycle walker for collaboration with a leader

Mohamad Shafiee Motahar
Sushant Veer
Ioannis Poulakakis

This paper presents a control method for steering three dimensional (3D) dynamically walking bipeds that are engaged in cooperative tasks such as object transportation. Towards achieving safe interaction with a leading human (or robot) collaborator, the walking biped is required to exhibit compliance at the port of interaction, while simultaneously adapting its walking pattern in response to the perceived interaction forces. To address these issues, we propose a method that fuses impedance control of the biped's arm with position control of its legs in a way that the biped adaptively modifies its stepping pattern according to the collaborator's intentions. The method is applied on a 3D bipedal robot that is driven in the workspace by a collaborator, whose intention is communicated to the biped through the interaction force.

Details

IROS Conference 2015 Conference Paper

Integrating dynamic walking and arm impedance control for cooperative transportation

Mohamad Shafiee Motahar
Sushant Veer
Jian Huang
Ioannis Poulakakis

This paper presents a method for integrating a cooperative manipulation task in the design of dynamic walking motions for an underactuated bipedal robot. Applications that involve physical interaction between a walking biped and a leading human (or robot) collaborator, require that the biped exhibits compliance at the port of interaction with the collaborator, while at the same time be capable of adjusting its stepping pattern in response to the interaction forces developed. To achieve these objectives, the proposed method combines impedance control of the biped's arm with position control of its legs in a way that the closed-loop system adapts its stepping pattern in accordance with the collaborator's intentions. The method is applied in the case of a bipedal robot model walking over flat ground and up and down stairs of known geometry under the influence of a trajectory that is unknown to the biped and represents the intention of a collaborator.

Details

IROS Conference 2015 Conference Paper

On the adaptation of dynamic walking to persistent external forcing using hybrid zero dynamics control

Sushant Veer
Mohamad Shafiee Motahar
Ioannis Poulakakis

This paper investigates the ability of dynamically walking bipeds to adapt their motion to persistent exogenous forcing. Applications that involve physical interactions between a bipedal robot and other robots (or humans), require that the robot adjust its stepping pattern in response to externally applied force signals. In our setting, an underactuated bipedal robot model walks under the influence of an external force. First, the hybrid zero dynamics method is used to design controllers that stabilize periodic walking motions in the absence of the external force. Then, conditions are derived analytically under which these (unforced) periodic gaits are modified to new (possibly aperiodic) stepping patterns that are consistent with the external force. It is deduced that underactuation holds the key to the ability of our model to adapt to external forces.

Details