Arrow Research search

Author name cluster

Sai Vemprala

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

17 papers
2 author rows

Possible papers

17

ICRA Conference 2024 Conference Paper

ConBaT: Control Barrier Transformer for Safe Robot Learning from Demonstrations

  • Yue Meng
  • Sai Vemprala
  • Rogerio Bonatti
  • Chuchu Fan
  • Ashish Kapoor

Large-scale self-supervised models have recently revolutionized our ability to perform a variety of tasks within the vision and language domains. However, using such models for autonomous systems is challenging because of safety requirements: besides executing correct actions, an autonomous agent must also avoid the high cost and potentially fatal critical mistakes. Traditionally, self-supervised training mainly focuses on imitating previously observed behaviors, and the training demonstrations carry no notion of which behaviors should be explicitly avoided. In this work, we propose Control Barrier Transformer (ConBaT), an approach that learns safe behaviors from demonstrations in a self-supervised fashion. ConBaT is inspired by the concept of control barrier functions in control theory and uses a causal transformer that learns to predict safe robot actions autoregressively using a critic that requires minimal safety data labeling. During deployment, we employ a lightweight online optimization to find actions that ensure future states lie within the learned safe set. We apply our approach to different simulated control tasks and show that our method results in safer control policies compared to other classical and learning-based methods such as imitation learning, reinforcement learning, and model predictive control.

Details

ICRA Conference 2023 Conference Paper

LATTE: LAnguage Trajectory TransformEr

  • Arthur Bucker
  • Luis F. C. Figueredo
  • Sami Haddadin
  • Ashish Kapoor
  • Shuang Ma
  • Sai Vemprala
  • Rogerio Bonatti

Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-specific solutions with little generalizability between hardware platforms, often with the use of static sets of target actions and commands. This work instead proposes a flexible language-based framework that allows a user to modify generic robotic trajectories. Our method leverages pre-trained language models (BERT and CLIP) to encode the user's intent and target objects directly from a free-form text input and scene images, fuses geometrical features generated by a transformer encoder network, and finally outputs trajectories using a transformer decoder, without the need of priors related to the task or robot information. We significantly extend our own previous work presented in [1] by expanding the trajectory parametrization space to 3D and velocity as opposed to just XY movements. In addition, we now train the model to use actual images of the objects in the scene for context (as opposed to textual descriptions), and we evaluate the system in a diverse set of scenarios beyond manipulation, such as aerial and legged robots. Our simulated and real-life experiments demonstrate that our transformer model can successfully follow human intent, modifying the shape and speed of trajectories within multiple environments. Codebase avail-able at: https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git.

Details

IROS Conference 2023 Conference Paper

PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training

  • Rogerio Bonatti
  • Sai Vemprala
  • Shuang Ma
  • Felipe Frujeri
  • Shuhang Chen
  • Ashish Kapoor

Robotics has long been a field riddled with complex systems architectures whose modules and connections, whether traditional or learning-based, require significant human expertise and prior knowledge. Inspired by large pre-trained language models, this work introduces a paradigm for pretraining a general purpose representation that can serve as a starting point for multiple tasks on a given robot. We present the Perception-Action Causal Transformer (PACT), a generative transformer-based architecture that aims to build representations directly from robot data in a self-supervised fashion. Through autoregressive prediction of states and actions over time, our model implicitly encodes dynamics and behaviors for a particular robot. Our experimental evaluation focuses on the domain of mobile agents, where we show that this robot-specific representation can function as a single starting point to achieve distinct tasks such as safe navigation, localization and mapping. We evaluate two form factors: a wheeled robot that uses a LiDAR sensor as perception input (MuSHR), and a simulated agent that uses first-person RGB images (Habitat). We show that finetuning small task-specific networks on top of the larger pretrained model results in significantly better performance compared to training a single model from scratch for all tasks simultaneously, and comparable performance to training a separate large model for each task independently. By sharing a common good-quality representation across tasks we can lower overall model capacity and speed up the real-time deployment of such systems.

Details

NeurIPS Conference 2022 Conference Paper

3DB: A Framework for Debugging Computer Vision Models

  • Guillaume Leclerc
  • Hadi Salman
  • Andrew Ilyas
  • Sai Vemprala
  • Logan Engstrom
  • Vibhav Vineet
  • Kai Xiao
  • Pengchuan Zhang

We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation. We demonstrate, through a wide range of use cases, that 3DB allows users to discover vulnerabilities in computer vision systems and gain insights into how models make decisions. 3DB captures and generalizes many robustness analyses from prior work, and enables one to study their interplay. Finally, we find that the insights generated by the system transfer to the physical world. 3DB will be released as a library alongside a set of examples and documentation. We attach 3DB to the submission.

PDF Details

CLeaR Conference 2022 Conference Paper

CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning

  • Daniel McDuff
  • Yale Song
  • Jiyoung Lee
  • Vibhav Vineet
  • Sai Vemprala
  • Nicholas Alexander Gyde
  • Hadi Salman
  • Shuang Ma

The ability to perform causal and counterfactual reasoning are central properties of human intelligence. Decision-making systems that can perform these types of reasoning have the potential to be more generalizable and interpretable. Simulations have helped advance the state-of-the-art in this domain, by providing the ability to systematically vary parameters (e. g. , confounders) and generate examples of the outcomes in the case of counterfactual scenarios. However, simulating complex temporal causal events in multi-agent scenarios, such as those that exist in driving and vehicle navigation, is challenging. To help address this, we present a high-fidelity simulation environment that is designed for developing algorithms for causal discovery and counterfactual reasoning in the safety-critical context. A core component of our work is to introduce agency, such that it is simple to define and create complex scenarios using high-level definitions. The vehicles then operate with agency to complete these objectives, meaning low-level behaviors need only be controlled if necessary. We perform experiments with three state-of-the-art methods to create baselines and highlight the affordances of this environment. Finally, we highlight challenges and opportunities for future work.

PDF Details

IROS Conference 2022 Conference Paper

COMPASS: Contrastive Multimodal Pretraining for Autonomous Systems

  • Shuang Ma
  • Sai Vemprala
  • Wenshan Wang
  • Jayesh K. Gupta
  • Yale Song
  • Daniel McDuff
  • Ashish Kapoor

Learning representations that generalize across tasks and domains is challenging yet necessary for autonomous systems. Although task-driven approaches are appealing, de-signing models specific to each application can be difficult in the face of limited data, especially when dealing with highly variable multimodal input spaces arising from different tasks in different environments. We introduce the first general-purpose pretraining pipeline, COntrastive Multimodal Pretraining for AutonomouS Systems (COMPASS), to overcome the limitations of task-specific models and existing pretraining approaches. COMPASS constructs a multimodal graph by considering the essential information for autonomous systems and the proper-ties of different modalities. Through this graph, multimodal signals are connected and mapped into two factorized spatio-temporal latent spaces: a “motion pattern space” and a “current state space. ” By learning from multimodal correspondences in each latent space, COMPASS creates state representations that models necessary information such as temporal dynamics, geometry, and semantics. We pretrain COMPASS on a large-scale multimodal simulation dataset TartanAir [1] and evaluate it on drone navigation, vehicle racing, and visual odometry tasks. The experiments indicate that COMPASS can tackle all three scenarios and can also generalize to unseen environments and real-world data. 1 1 Our code implementation can be found at https://github.com/microsoft/COMPASS

Details

NeurIPS Conference 2022 Conference Paper

Learning Modular Simulations for Homogeneous Systems

  • Jayesh Gupta
  • Sai Vemprala
  • Ashish Kapoor

Complex systems are often decomposed into modular subsystems for engineering tractability. Although various equation based white-box modeling techniques make use of such structure, learning based methods have yet to incorporate these ideas broadly. We present a modular simulation framework for modeling homogeneous multibody dynamical systems, which combines ideas from graph neural networks and neural differential equations. We learn to model the individual dynamical subsystem as a neural ODE module. Full simulation of the composite system is orchestrated via spatio-temporal message passing between these modules. An arbitrary number of modules can be combined to simulate systems of a wide variety of coupling topologies. We evaluate our framework on a variety of systems and show that message passing allows coordination between multiple modules over time for accurate predictions and in certain cases, enables zero-shot generalization to new system configurations. Furthermore, we show that our models can be transferred to new system configurations with lower data requirement and training effort, compared to those trained from scratch.

PDF Details

IROS Conference 2022 Conference Paper

Learning to Simulate Realistic LiDARs

  • Benoît Guillard
  • Sai Vemprala
  • Jayesh K. Gupta
  • Ondrej Miksik
  • Vibhav Vineet
  • Pascal Fua
  • Ashish Kapoor

Simulating realistic sensors is a challenging part in data generation for autonomous systems, often involving carefully handcrafted sensor design, scene properties, and physics modeling. To alleviate this, we introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We propose a model that learns a mapping between RGB images and corresponding LiDAR features such as raydrop or perpoint intensities directly from real datasets. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces or high intensity returns on reflective materials. When applied to naively raycasted point clouds provided by off-the-shelf simulator software, our model enhances the data by predicting intensities and removing points based on the scene's appearance to match a real LiDAR sensor. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly. Through a sample task of vehicle segmentation, we show that enhancing simulated point clouds with our technique improves downstream task performance.

Details

ICLR Conference 2022 Conference Paper

Missingness Bias in Model Debugging

  • Saachi Jain
  • Hadi Salman
  • Eric Wong 0001
  • Pengchuan Zhang
  • Vibhav Vineet
  • Sai Vemprala
  • Aleksander Madry

Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools. However, in computer vision, pixels cannot simply be removed from an image. One thus tends to resort to heuristics such as blacking out pixels, which may in turn introduce bias into the debugging process. We study such biases and, in particular, show how transformer-based architectures can enable a more natural implementation of missingness, which side-steps these issues and improves the reliability of model debugging in practice.

Details

ICRA Conference 2021 Conference Paper

Adversarial Attacks on Optimization based Planners

  • Sai Vemprala
  • Ashish Kapoor

Trajectory planning is a key piece in the algorithmic architecture of a robot. Trajectory planners typically use iterative optimization schemes for generating smooth trajectories that avoid collisions and are optimal for tracking given the robot’s physical specifications. Starting from an initial estimate, the planners iteratively refine the solution so as to satisfy the desired constraints. In this paper, we show that such iterative optimization based planners can be vulnerable to adversarial attacks that force the planner either to fail completely, or significantly increase the time required to find a solution. The key insight here is that an adversary in the environment can directly affect the optimization cost function of a planner. We demonstrate how the adversary can adjust its own state configurations to result in poorly conditioned eigenstructure of the objective leading to failures. We apply our method against two state of the art trajectory planners and demonstrate that an adversary can consistently exploit certain weaknesses of an iterative optimization scheme.

Details

NeurIPS Conference 2021 Conference Paper

Representation Learning for Event-based Visuomotor Policies

  • Sai Vemprala
  • Sami Mian
  • Ashish Kapoor

Event-based cameras are dynamic vision sensors that provide asynchronous measurements of changes in per-pixel brightness at a microsecond level. This makes them significantly faster than conventional frame-based cameras, and an appealing choice for high-speed robot navigation. While an interesting sensor modality, this asynchronously streamed event data poses a challenge for machine learning based computer vision techniques that are more suited for synchronous, frame-based data. In this paper, we present an event variational autoencoder through which compact representations can be learnt directly from asynchronous spatiotemporal event data. Furthermore, we show that such pretrained representations can be used for event-based reinforcement learning instead of end-to-end reward driven perception. We validate this framework of learning event-based visuomotor policies by applying it to an obstacle avoidance scenario in simulation. Compared to techniques that treat event data as images, we show that representations learnt from event streams result in faster policy training, adapt to different control capacities, and demonstrate a higher degree of robustness to environmental changes and sensor noise.

PDF Details

NeurIPS Conference 2021 Conference Paper

Unadversarial Examples: Designing Objects for Robust Vision

  • Hadi Salman
  • Andrew Ilyas
  • Logan Engstrom
  • Sai Vemprala
  • Aleksander Madry
  • Ashish Kapoor

We study a class of computer vision settings wherein one can modify the design of the objects being recognized. We develop a framework that leverages this capability---and deep networks' unusual sensitivity to input perturbations---to design ``robust objects, '' i. e. , objects that are explicitly optimized to be confidently classified. Our framework yields improved performance on standard benchmarks, a simulated robotics environment, and physical-world experiments.

PDF Details

IROS Conference 2020 Conference Paper

Safety Considerations in Deep Control Policies with Safety Barrier Certificates Under Uncertainty

  • Tom Hirshberg
  • Sai Vemprala
  • Ashish Kapoor

Recent advances in Deep Machine Learning have shown promise in solving complex perception and control loops via methods such as reinforcement and imitation learning. However, guaranteeing safety for such learned deep policies has been a challenge due to issues such as partial observability and difficulties in characterizing the behavior of the neural networks. While a lot of emphasis in safe learning has been placed during training, it is non-trivial to guarantee safety at deployment or test time. This paper extends how under mild assumptions, Safety Barrier Certificates can be used to guarantee safety with deep control policies despite uncertainty arising due to perception and other latent variables. Specifically for scenarios where the dynamics are smooth and uncertainty has a finite support, the proposed framework wraps around an existing deep control policy and generates safe actions by dynamically evaluating and modifying the policy from the embedded network. Our framework utilizes control barrier functions to create spaces of control actions that are safe under uncertainty, and when the original actions are found to be in violation of the safety constraint, uses quadratic programming to minimally modify the original actions to ensure they lie in the safe set. Representations of the environment are built through Euclidean signed distance fields that are then used to infer the safety of actions and to guarantee forward invariance. We implement this method in simulation in a drone-racing environment and show that our method results in safer actions compared to a baseline that only relies on imitation learning to generate control actions.

Details

IROS Conference 2018 Conference Paper

Drone Detection Using Depth Maps

  • Adrian Carrio
  • Sai Vemprala
  • Andres Ripoll
  • Srikanth Saripalli
  • Pascual Campoy

Obstacle avoidance is a key feature for safe Unmanned Aerial Vehicle (UAV) navigation. While solutions have been proposed for static obstacle avoidance, systems enabling avoidance of dynamic objects, such as drones, are hard to implement due to the detection range and field-of-view (FOV) requirements, as well as the constraints for integrating such systems on-board small UAVs. In this work, a dataset of 6k synthetic depth maps of drones has been generated and used to train a state-of-the-art deep learning-based drone detection model. While many sensing technologies can only provide relative altitude and azimuth of an obstacle, our depth map-based approach enables full 3D localization of the obstacle. This is extremely useful for collision avoidance, as 3D localization of detected drones is key to perform efficient collision-free path planning. The proposed detection technique has been validated in several real depth map sequences, with multiple types of drones flying at up to 2 m/s, achieving an average precision of 98. 7 %, an average recall of 74. 7 % and a record detection range of 9. 5 meters.

Details

IROS Conference 2018 Conference Paper

Real-Time Tumor Tracking for Pencil Beam Scanning Proton Therapy

  • Sai Vemprala
  • Srikanth Saripalli
  • Carlos E. Vargas
  • Martin Bues
  • Yanle Hu
  • Jiajian Shen

In this paper, we describe the method and implementation of a real-time tumor tracking system for a pencil beam scanning (PBS) proton therapy system. PBS is an advanced cancer treatment system that can benefit from precise localization of the tumors through motion. We utilize techniques such as cross-correlation matching, correlation filters and small object saliency, creating an array of methods that can detect and track fiducial markers implanted in the cancer tumors. The final aim is to control the proton beam using real-time image guidance. Our technique works robustly on various types of markers such as ceramic/metallic fiducials, visicoil markers and surgical clips. Left and right views of an X-ray fluoroscopy system were utilized to also triangulate the marker positions in full 3D as they are tracked through normal breathing movement and organ motion. We have tested our detection system on data from several patients with different tumor locations both offline and in real-time and wish to implement it within a full treatment system soon. To the best of the authors knowledge, this is the first real time tracking system for PBS therapy that is applicable for various types of fiducials and tumor locations.

Details

ICRA Conference 2018 Conference Paper

Vision Based Collaborative Path Planning for Micro Aerial Vehicles

  • Sai Vemprala
  • Srikanth Saripalli

In this paper, we present a collaborative path-planning framework for a group of micro aerial vehicles that are capable of localizing through vision. Each of the micro aerial vehicles is assumed to be equipped with a forward facing monocular camera. The vehicles initially use their captured images to build 3D maps through common features; and subsequently track these features to localize through 3D-2D correspondences. The planning algorithm, while connecting start locations to provided goal locations, also aims to reduce the localization uncertainty of the vehicles in the group. To achieve this, we develop a two-step planning framework: the first step attempts to build an improved map of the environment by solving the next-best-view problem for multiple cameras. We express this as a black-box optimization problem and solve it using the Covariance Matrix Adaption evolution strategy (CMA-ES). Once an improved map is available, the second stage of the planning framework performs belief space planning for the vehicles individually using the rapidly exploring random belief tree (RRBT) algorithm. Through the RRBT approach, the planner generates paths that ensure feature visibility while attempting to optimize path cost and reduce localization uncertainty. We validate our approach using experiments conducted in a high visual-fidelity aerial vehicle simulator, Microsoft AirSim.

Details

IROS Conference 2016 Conference Paper

Vision based collaborative localization for multirotor vehicles

  • Sai Vemprala
  • Srikanth Saripalli

We present a framework for vision based localization for two or more multirotor aerial vehicles relative to each other. This collaborative localization technique is built upon a relative pose estimation strategy between two or more cameras with the capability of estimating accurate metric poses between each other even through fast motion and continually changing environments. Through synchronized feature detection and tracking with a robust outlier rejection process, classical multiple view geometry concepts have been utilized for obtaining scale-ambiguous relative poses, which are then refined through reconstruction and pose optimization to provide a metric estimate. Furthermore, we present the implementation details of this technique followed by a set of results which involves evaluation of the accuracy of the pose estimates through test cases in both simulated and real experiments. Test cases include keeping one camera stationary as the other is mounted on a quadrotor which is then flown through various types of trajectories. We also perform a quantitative comparison with a GPS/IMU localization technique to demonstrate the accuracy of our method.

Details
v2026.09.13