Author name cluster

Adam Fishman

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

1 author row

ICRA Conference 2025 Conference Paper

Cascaded Diffusion Models for Neural Motion Planning

Mohit Sharma 0001
Adam Fishman
Vikash Kumar
Chris Paxton 0001
Oliver Kroemer

Robots in the real world need to perceive and move to goals in complex environments without collisions. Avoiding collisions is especially difficult when relying on sensor perception and when goals are among clutter. Diffusion policies and other generative models have shown strong performance in solving local planning problems, but often struggle at avoiding all of the subtle constraint violations that characterize truly challenging global motion planning problems. In this work, we propose an approach for learning global motion planning using diffusion policies, allowing the robot to generate full trajectories through complex scenes and reasoning about multiple obstacles along the path. Our approach uses cascaded hierarchical models which unify global prediction and local refinement together with online plan repair to ensure the trajectories are collision free. Our method outperforms ( $\approx 5 \%$ ) a wide variety of baselines on challenging tasks in multiple domains including navigation and manipulation.

Details

ICRA Conference 2025 Conference Paper

This&That: Language-Gesture Controlled Video Generation for Robot Planning

Boyang Wang
Nikhil Sridhar
Chao Feng
Mark Van der Merwe
Adam Fishman
Nima Fazeli
Jeong Joon Park

Clear, interpretable instructions are invaluable for complex tasks, helping to clarify goals and anticipate necessary steps. In this work, we propose a robot learning framework for communicating, planning, and executing a wide range of tasks, dubbed This&That. This&That solves general tasks by leveraging video generative models, which, through training on internet-scale data, contain rich physical and semantic context. Through this work, we tackle three fundamental challenges in video-based planning: 1) unambiguous task communication with simple human instructions, 2) controllable video gen-eration that respects user intent, and 3) translating visual plans into robot actions. This& That adds gesture conditioning alongside language to generate video predictions as a suc-cinct and unambiguous alternative to existing language-only methods, especially in complex and uncertain environments. These video predictions are then fed into a behavior cloning architecture dubbed Diffusion Video to Action (DiVA), which outperforms prior state-of-the-art behavior cloning and video-based planning methods by substantial margins. Project web-site: https://this-and-that-vid.github.io/this-and-thatl.

Details

ICRA Conference 2023 Conference Paper

CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generation

Adithyavairavan Murali
Arsalan Mousavian
Clemens Eppner
Adam Fishman
Dieter Fox

We address the important problem of generalizing robotic rearrangement to clutter without any explicit object models. We first generate over 650K cluttered scenes-orders of magnitude more than prior work-in diverse everyday environments, such as cabinets and shelves. We render synthetic partial point clouds from this data and use it to train our CabiNet model architecture. CabiNet is a collision model that accepts object and scene point clouds, captured from a single-view depth observation, and predicts collisions for SE(3) object poses in the scene. Our representation has a fast inference speed of 7μs/query with nearly 20% higher performance than baseline approaches in challenging environments. We use this collision model in conjunction with a Model Predictive Path Integral (MPPI) planner to generate collision-free trajectories for picking and placing in clutter. CabiNet also predicts waypoints, computed from the scene's signed distance field (SDF), that allows the robot to navigate tight spaces during rearrangement. This improves rearrangement performance by nearly 35% compared to baselines. We systematically evaluate our approach, procedurally generate simulated experiments, and demonstrate that our approach directly transfers to the real world, despite training exclusively in simulation. Supplementary material and videos of robot experiments in completely unknown scenes are available at: cabinet-object-rearrangement.github.io.

Details

ICRA Conference 2023 Conference Paper

CuRobo: Parallelized Collision-Free Robot Motion Generation

Balakumar Sundaralingam
Siva Kumar Sastry Hari
Adam Fishman
Caelan Reed Garrett
Karl Van Wyk
Valts Blukis
Alexander Millane
Helen Oleynikova

This paper explores the problem of collision-free motion generation for manipulators by formulating it as a global motion optimization problem. We develop a parallel optimization technique to solve this problem and demonstrate its effectiveness on massively parallel GPUs. We show that combining simple optimization techniques with many parallel seeds leads to solving difficult motion generation problems within 53ms on average, 62x faster than SOTA trajectory optimization methods. We achieve SOTA performance by combining L-BFGS step direction estimation with a novel parallel noisy line search scheme and a particle-based optimization solver. To further aid trajectory optimization, we develop a parallel geometric planner that is atleast 28x faster than SOTA RRTConnect implementations. We also introduce a collision-free IK solver that can solve over 9000 queries/s. We are releasing our GPU accelerated library CuRobo that contains core components for robot motion generation. Additional details are available at sites.google.com/nvidia.com/curobo.

Details

IROS Conference 2021 Conference Paper

Reactive Long Horizon Task Execution via Visual Skill and Precondition Models

Shohin Mukherjee
Chris Paxton 0001
Arsalan Mousavian
Adam Fishman
Maxim Likhachev
Dieter Fox

Zero-shot execution of unseen robotic tasks is important to allowing robots to perform a wide variety of tasks in human environments, but collecting the amounts of data necessary to train end-to-end policies in the real-world is often infeasible. We describe an approach for sim-to-real training that can accomplish unseen robotic tasks using models learned in simulation to ground components of a simple task planner. We learn a library of parameterized skills, along with a set of predicates-based preconditions and termination conditions, entirely in simulation. We explore a block-stacking task because it has a clear structure, where multiple skills must be chained together, but our methods are applicable to a wide range of other problems and domains, and can transfer from simulation to the real-world with no fine tuning. The system is able to recognize failures and accomplish long-horizon tasks from perceptual input, which is critical for real-world execution. We evaluate our proposed approach in both simulation and in the real-world, showing an increase in success rate from 91. 6% to 98% in simulation and from 10% to 80% success rate in the real-world as compared with naive baselines. For experiment videos including both real-world and simulation, see: https://www.youtube.com/playlist? list=PL-oD0xHUngeLfQmpngYkGFZarstfPOXqX

Details

IROS Conference 2020 Conference Paper

Collaborative Interaction Models for Optimized Human-Robot Teamwork

Adam Fishman
Chris Paxton 0001
Wei Yang 0019
Dieter Fox
Byron Boots
Nathan D. Ratliff

Effective human-robot collaboration requires informed anticipation. The robot must anticipate the human’s actions, but also react quickly and intuitively when its predictions are wrong. The robot must plan its actions to account for the human’s own plan, with the knowledge that the human’s behavior will change based on what the robot actually does. This cyclical game of predicting a human’s future actions and generating a corresponding motion plan is extremely difficult to model using standard techniques. In this work, we describe a novel Model Predictive Control (MPC)-based framework for finding optimal trajectories in a collaborative, multi-agent setting, in which we simultaneously plan for the robot while predicting the actions of its external collaborators. We use human-robot handovers to demonstrate that with a strong model of the collaborator, our framework produces fluid, reactive human-robot interactions in novel, cluttered environments. Our method efficiently generates coordinated trajectories, and achieves a high success rate in handover, even in the presence of significant sensor noise.

Details