Author name cluster

Dmitry Berenson

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

38 papers

1 author row

ICRA Conference 2025 Conference Paper

Diffusion-Informed Probabilistic Contact Search for Multi-Finger Manipulation

Abhinav Kumar
Thomas Power
Fan Yang 0144
Sergio Aguilera Marinovic
Soshi Iba
Rana Soltani-Zarrin
Dmitry Berenson

Planning contact-rich interactions for multi-finger manipulation is challenging due to the high-dimensionality and hybrid nature of dynamics. Recent advances in data-driven methods have shown promise, but are sensitive to the quality of training data. Combining learning with classical methods like trajectory optimization and search adds additional structure to the problem and domain knowledge in the form of constraints, which can lead to outperforming the data on which models are trained. We present Diffusion-Informed Probabilistic Contact Search (DIPS), which uses an A* search to plan a sequence of contact modes informed by a diffusion model. We train the diffusion model on a dataset of demonstrations consisting of contact modes and trajectories generated by a trajectory optimizer given those modes. In addition, we use a particle filter-inspired method to reason about variability in diffusion sampling arising from model error, estimating likelihoods of trajectories using a learned discriminator. We show that our method outperforms ablations that do not reason about variability and can plan contact sequences that outperform those found in training data across multiple tasks. We evaluate on simulated tabletop card sliding and screwdriver turning tasks, as well as the screwdriver task in hardware to show that our combined learning and planning approach transfers to the real world.

ICRA Conference 2025 Conference Paper

Implicit Contact Diffuser: Sequential Contact Reasoning With Latent Point Cloud Diffusion

Zixuan Huang
Yinong He
Yating Lin
Dmitry Berenson

Long-horizon contact-rich manipulation has long been a challenging problem, as it requires reasoning over both discrete contact modes and continuous object motion. We introduce Implicit Contact Diffuser (ICD), a diffusion-based model that generates a sequence of neural descriptors that specify a series of contact relationships between the object and the environment. This sequence is then used as guidance for an MPC method to accomplish a given task. The key advantage of this approach is that the latent descriptors provide more taskrelevant guidance to MPC, helping to avoid local minima for contact-rich manipulation tasks. Our experiments demonstrate that ICD outperforms baselines on complex, long-horizon, contact-rich manipulation tasks, such as cable routing and notebook folding. Additionally, our experiments also indicate that ICD can generalize a target contact relationship to a different environment. More visualizations can be found on our website https://implicit-contact-diffuser.github.io.

ICRA Conference 2025 Conference Paper

Language-Guided Object Search in Agricultural Environments

Advaith Balaji
Saket Pradhan
Dmitry Berenson

Creating robots that can assist in farms and gardens can help reduce the mental and physical workload experienced by farm workers. We tackle the problem of object search in a farm environment, providing a method that allows a robot to semantically reason about the location of an unseen target object among a set of previously seen objects in the environment using a Large Language Model (LLM). We leverage object-to-object semantic relationships to plan a path through the environment that will allow us to accurately and efficiently locate our target object while also reducing the overall distance traveled, without needing high-level room or area-level semantic relationships. During our evaluations, we found that our method outperformed a current state-of-the-art baseline and our ablations. Our offline testing yielded an average path efficiency of 84 %, reflecting how closely the predicted path aligns with the ideal path. Upon deploying our system on the Boston Dynamics Spot robot in a real-world farm environment, we found that our system had a success rate of 80 %, with a success weighted by path length of 0. 67, which demonstrates a reasonable trade-off between task success and path efficiency under real-world conditions. The project website can be viewed at: adi-balaji.github.io/losae

ICRA Conference 2024 Conference Paper

Improving Out-of-Distribution Generalization of Learned Dynamics by Learning Pseudometrics and Constraint Manifolds

Yating Lin
Glen Chou
Dmitry Berenson

We propose a method for improving the prediction accuracy of learned robot dynamics models on out-of-distribution (OOD) states. We achieve this by leveraging two key sources of structure often present in robot dynamics: 1) sparsity, i. e. , some components of the state may not affect the dynamics, and 2) physical limits on the set of possible motions, in the form of nonholonomic constraints. Crucially, we do not assume this structure is known a priori, and instead learn it from data. We use contrastive learning to obtain a distance pseudometric that uncovers the sparsity pattern in the dynamics, and use it to reduce the input space when learning the dynamics. We then learn the unknown constraint manifold by approximating the normal space of possible motions from the data, which we use to train a Gaussian process (GP) representation of the constraint manifold. We evaluate our approach on a physical differential-drive robot and a simulated quadrotor, showing improved prediction accuracy on OOD data relative to baselines.

ICRA Conference 2024 Conference Paper

Online Adaptation of Sampling-Based Motion Planning with Inaccurate Models

Marco Faroni
Dmitry Berenson

Robotic manipulation relies on analytical or learned models to simulate the system dynamics. These models are often inaccurate and based on offline information, so that the robot planner is unable to cope with mismatches between the expected and the actual behavior of the system (e. g. , the presence of an unexpected obstacle). In these situations, the robot should use information gathered online to correct its planning strategy and adapt to the actual system response. We propose a sampling-based motion planning approach that uses an estimate of the model error and online observations to correct the planning strategy at each new replanning. Our approach adapts the cost function and the sampling bias of a kinodynamic motion planner when the outcome of the executed transitions is different from the expected one (e. g. , when the robot unexpectedly collides with an obstacle) so that future trajectories will avoid unreliable motions. To infer the properties of a new transition, we introduce the notion of context-awareness, i. e. , we store local environment information for each executed transition and avoid new transitions with context similar to previous unreliable ones. This is helpful for leveraging online information even if the simulated transitions are far (in the state-and-action space) from the executed ones. Simulation and experimental results show that the proposed approach increases the success rate in execution and reduces the number of replannings needed to reach the goal.

ICRA Conference 2024 Conference Paper

Subgoal Diffuser: Coarse-to-fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulation

Zixuan Huang
Yating Lin
Fan Yang 0144
Dmitry Berenson

Manipulation of articulated and deformable objects can be difficult due to their compliant and under-actuated nature. Unexpected disturbances can cause the object to deviate from a predicted state, making it necessary to use Model-Predictive Control (MPC) methods to plan motion. However, these methods need a short planning horizon to be practical. Thus, MPC is ill-suited for long-horizon manipulation tasks due to local minima. In this paper, we present a diffusion-based method that guides an MPC method to accomplish long-horizon manipulation tasks by dynamically specifying sequences of subgoals for the MPC to follow. Our method, called Subgoal Diffuser, generates subgoals in a coarse-to-fine manner, producing sparse subgoals when the task is easily accomplished by MPC and more dense subgoals when the MPC method needs more guidance. The density of subgoals is determined dynamically based on a learned estimate of reachability, and subgoals are distributed to focus on challenging parts of the task. We evaluate our method on two robot manipulation tasks and find it improves the planning performance of an MPC method, and also outperforms prior diffusion-based methods. More visualizations and results can be found at https://sites.google.com/view/subgoal-diffuser-mpc

ICRA Conference 2024 Conference Paper

The Grasp Loop Signature: A Topological Representation for Manipulation Planning with Ropes and Cables

Peter Mitrano
Dmitry Berenson

This paper studies robotic manipulation of deformable, one-dimensional objects (DOOs) like ropes or cables, which has important potential applications in manufacturing, agriculture, and surgery. In such environments, the task may involve threading through or avoiding becoming tangled with other objects. Grasping with multiple grippers can create closed loops between the robot and DOO, and if an obstacle lies within this loop, it may be impossible to reach the goal. However, prior work has only considered the topology of the DOO in isolation, ignoring the arms that are manipulating it. Searching over possible grasps to accomplish the task without considering such topological information is very inefficient, as many grasps will not lead to progress on the task due to topological constraints. Therefore, we propose the ${{\mathcal{G}}_L} - {\text{signature}}$ which categorizes the topology of these grasp loops and show how it can be used to guide planning. We perform experiments in simulation on two DOO manipulation tasks to show that using the ${{\mathcal{G}}_L} - {\text{signature}}$ is faster and more successful than methods that rely on local geometry or additional finite-horizon planning. Finally, we demonstrate using the ${{\mathcal{G}}_L} - {\text{signature}}$ in a real-world dual-arm cable manipulation task.

ICRA Conference 2023 Conference Paper

Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification

Jiayi Pan 0002
Glen Chou
Dmitry Berenson

To make robots accessible to a broad audience, it is critical to endow them with the ability to take universal modes of communication, like commands given in natural language, and extract a concrete desired task specification, defined using a formal language like linear temporal logic (LTL). In this paper, we present a learning-based approach for translating from natural language commands to LTL specifications with very limited human-labeled training data. This is in stark contrast to existing natural-language to LTL translators, which require large human-labeled datasets, often in the form of labeled pairs of LTL formulas and natural language commands, to train the translator. To reduce reliance on human data, our approach generates a large synthetic training dataset through algorithmic generation of LTL formulas, conversion to structured English, and then exploiting the paraphrasing capabilities of modern large language models (LLMs) to synthesize a diverse corpus of natural language commands corresponding to the LTL formu-las. We use this generated data to finetune an LLM and apply a constrained decoding procedure at inference time to ensure the returned LTL formula is syntactically correct. We evaluate our approach on three existing LTL/natural language datasets and show that we can translate natural language commands at 75% accuracy with far less human data (≤12 annotations). Moreover, when training on large human-annotated datasets, our method achieves higher test accuracy (95% on average) than prior work. Finally, we show the translated formulas can be used to plan long-horizon, multi-stage tasks on a 12D quadrotor.

ICRA Conference 2023 Conference Paper

Focused Adaptation of Dynamics Models for Deformable Object Manipulation

Peter Mitrano
Alex LaGrassa
Oliver Kroemer
Dmitry Berenson

In order to efficiently learn a dynamics model for a task in a new environment, one can adapt a model learned in a similar source environment. However, existing adaptation methods can fail when the target dataset contains transitions where the dynamics are very different from the source environment. For example, the source environment dynamics could be of a rope manipulated in free space, whereas the target dynamics could involve collisions and deformation on obstacles. Our key insight is to improve data efficiency by focusing model adaptation on only the regions where the source and target dynamics are similar. In the rope example, adapting the free-space dynamics requires significantly less data than adapting the free-space dynamics while also learning collision dynamics. We propose a new method for adaptation that is effective in adapting to regions of similar dynamics. Additionally, we combine this adaptation method with prior work on planning with unreliable dynamics to make a method for data-efficient online adaptation, called FOCUS. We first demonstrate that the proposed adaptation method achieves statistically significantly lower prediction error in regions of similar dynamics on simulated rope manipulation and plant watering tasks. We then show on a bimanual rope manipulation task that FOCUS achieves data-efficient online learning, in simulation and in the real world.

ICRA Conference 2021 Conference Paper

Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation

Andrew Price
Kun Huang
Dmitry Berenson

Despite rapid progress in scene segmentation in recent years, 3D segmentation methods are still limited when there is severe occlusion. The key challenge is estimating the segment boundaries of (partially) occluded objects, which are inherently ambiguous when considering only a single frame. In this work, we propose Multihypothesis Segmentation Tracking (MST), a novel method for volumetric segmentation in changing scenes, which allows scene ambiguity to be tracked and our estimates to be adjusted over time as we interact with the scene. Two main innovations allow us to tackle this difficult problem: 1) A novel way to sample possible segmentations from a segmentation tree; and 2) A novel approach to fusing tracking results with multiple segmentation estimates. These methods allow MST to track the segmentation state over time and incorporate new information, such as new objects being revealed. We evaluate our method on several cluttered tabletop environments in simulation and reality. Our results show that MST outperforms baselines in all tested scenes.

ICRA Conference 2021 Conference Paper

Tracking Partially-Occluded Deformable Objects while Enforcing Geometric Constraints

Yixuan Wang 0003
Dale McConachie
Dmitry Berenson

In order to manipulate a deformable object, such as rope or cloth, in unstructured environments, robots need a way to estimate its current shape. However, tracking the shape of a deformable object can be challenging because of the object’s high flexibility, (self-)occlusion, and interaction with obstacles. Building a high-fidelity physics simulation to aid in tracking is difficult for novel environments. Instead we focus on tracking the object based on RGBD images and geometric motion estimates and obstacles. Our key contributions over previous work in this vein are: 1) A better way to handle severe occlusion by using a motion model to regularize the tracking estimate; and 2) The formulation of convex geometric constraints, which allow us to prevent self-intersection and penetration into known obstacles via a post-processing step. These contributions allow us to outperform previous methods by a large margin in terms of accuracy in scenarios with severe occlusion and obstacles.

ICRA Conference 2019 Conference Paper

Efficient Humanoid Contact Planning using Learned Centroidal Dynamics Prediction

Yu-Chi Lin
Brahayam Ponton
Ludovic Righetti
Dmitry Berenson

Humanoid robots dynamically navigate an environment by interacting with it via contact wrenches exerted at intermittent contact poses. Therefore, it is important to consider dynamics when planning a contact sequence. Traditional contact planning approaches assume a quasi-static balance criterion to reduce the computational challenges of selecting a contact sequence over a rough terrain. This however limits the applicability of the approach when dynamic motions are required, such as when walking down a steep slope or crossing a wide gap. Recent methods overcome this limitation with the help of efficient mixed integer convex programming solvers capable of synthesizing dynamic contact sequences. Nevertheless, its exponential-time complexity limits its applicability to short time horizon contact sequences within small environments. In this paper, we go beyond current approaches by learning a prediction of the dynamic evolution of the robot centroidal momenta, which can then be used for quickly generating dynamically robust contact sequences for robots with arms and legs using a search-based contact planner. We demonstrate the efficiency and quality of the results of the proposed approach in a set of dynamically challenging scenarios.

IROS Conference 2019 Conference Paper

Occlusion-robust Deformable Object Tracking without Physics Simulation

Cheng Chi 0001
Dmitry Berenson

Estimating the state of a deformable object is crucial for robotic manipulation, yet accurate tracking is challenging when the object is partially-occluded. To address this problem, we propose an occlusion-robust RGBD sequence tracking framework based on Coherent Point Drift (CPD). To mitigate the effects of occlusion, our method 1) Uses a combination of locally linear embedding and constrained optimization to regularize the output of CPD, thus enforcing topological consistency when occlusions create disconnected pieces of the object; 2) Reasons about the free-space visible by an RGBD sensor to better estimate the prior on point location and to detect tracking failures during occlusion; and 3) Uses shape descriptors to find the most relevant previous state of the object to use for tracking after a severe occlusion. Our method does not rely on physics simulation or a physical model of the object, which can be difficult to obtain in unstructured environments. Despite having no physical model, our experiments demonstrate that our method achieves improved accuracy in the presence of occlusion as compared to a physics-based CPD method while maintaining adequate run-time.

IROS Conference 2018 Conference Paper

Accounting for Directional Rigidity and Constraints in Control for Manipulation of Deformable Objects without Physical Simulation

Mengyao Ruan
Dale McConachie
Dmitry Berenson

Deformable objects like cloth and rope are challenging to manipulate because it is difficult to predict the state of the object given a motion of the gripper(s) holding it. In much previous work, physical models (such as Mass-Spring or Finite-Element) have been used to model such affects. However, these models often require significant parameter tuning for each scenario and can be expensive to simulate inside a control loop. Furthermore, it is difficult to create a practical controller for deformable object manipulation that preserves constraints, especially avoiding overstretching the object. In this paper, we developed a more effective controller than previous work by (1) constructing a more accurate geometric model of how the direction of gripper motion and obstacles affect deformable objects; and (2) specifying a novel stretching avoidance constraint to prevent the object from being overstretched by the robot. Experiments comparing our new method to the previous method in simulation and on a physical robot suggest that our new model captures the behavior of the object more accurately. We also find that our controller is able to prevent tearing that would occur when using the previous method.

IROS Conference 2018 Conference Paper

Humanoid Navigation Planning in Large Unstructured Environments Using Traversability - Based Segmentation

Yu-Chi Lin
Dmitry Berenson

Humanoids' abilities to navigate stairs and uneven terrain make them well-suited for disaster response efforts. However, humanoid navigation in such environments is currently limited by the capabilities of navigation planners. Such planners typically consider only footstep locations, but planning with palm contacts may be necessary to cross a gap, avoid an obstacle, or maintain balance. However, considering palm contacts greatly increases the branching factor of the search, leading to impractical planning times for large environments. In previous work we explored using library-based methods to address difficult navigation planning problems requiring palm contacts, but such methods are not efficient when navigating an easy-to-traverse part of the environment. To maximize planning efficiency, we would like to use discrete planners when an area is easy to traverse and switch to the library-based method only when traversal becomes difficult. Thus, in this paper we present a method that 1) Plans a guiding torso path which accounts for the difficulty of traversing the environment as predicted by learned regressors; and 2) Decomposes the guiding path into a set of segments, each of which is assigned a motion mode (i. e. a set of feet and hands to use) and a planning method. Easily-traversable segments are assigned a discrete-search planner, while other segments are assigned a library-based method that fits existing motion plans to the environment near the given segment. Our results suggest that this segmentation approach greatly outperforms standard discrete planning and that using the library-based method for more difficult segments gives a benefit over using discrete planning.

ICRA Conference 2017 Conference Paper

A framework for robot-assisted doffing of personal protective equipment

Antonio Umali
Dmitry Berenson

When treating highly-infectious diseases such as Ebola, health workers are at high risk of infection during the doffing of Personal Protective Equipment (PPE). This is due to factors such as fatigue, hastiness, and inconsistency in training. The introduction of a semi-autonomous robot doffing assistant has the potential to increase the safety of the doffing procedure by assisting the human during high-risk sub-tasks. However, using a robotic assistant requires transforming a purely human task into a sequence of safe and effective human-robot collaborative actions. Since diseases like Ebola can spread through the mucous membranes of the face our goal in synthesizing these actions is to keep the human's hands away from his or her face as much as possible. As a secondary goal, we also seek to minimize the human's effort. We segment the doffing procedure into a sequence of human and robot actions such that the robot only assists when necessary and the human performs the more intricate parts of the procedure. Our framework then synthesizes assistive motions for the robot that perform parts of the tasks. Our experiments on five doffing tasks suggest that the introduction of a robot assistant improves the safety of the procedure in three out of four of the high-risk doffing tasks while reducing effort in all five tasks.

ICRA Conference 2016 Conference Paper

Considering avoidance and consistency in motion planning for human-robot manipulation in a shared workspace

Rafi Hayne
Ruikun Luo
Dmitry Berenson

This paper presents an approach to formulating the cost function for a motion planner intended for human-robot collaboration on manipulation tasks in a shared workspace. To be effective for human-robot collaboration a robot should plan its motion so that it is both safe and efficient. To achieve this, we propose two factors to consider in the cost function for the robot's motion planner: (1) Avoidance of the workspace previously-occupied by the human, so that the motion is as safe as possible, and (2) Consistency of the robot's motion, so that the motion is as predictable as possible for the human and they can perform their task without focusing undue attention on the robot. Our experiments in simulation and a human-robot workspace sharing study compare a cost function that uses only the first factor and a combined cost that uses both factors vs. a baseline method that is perfectly consistent but does not account for the human's previous motion. We find that using either cost function we outperform the baseline method in terms of task success rate without degrading the task completion time. The best task success rate is achieved with the cost function that includes both the avoidance and consistency terms.

ICRA Conference 2016 Conference Paper

Improving Soft Pneumatic Actuator fingers through integration of soft sensors, position and force control, and rigid fingernails

John Morrow
Hee-Sup Shin
Calder Phillips-Grafflin
Sung-Hwan Jang
Jacob Torrey
Riley Larkins
Steven Dang
Yong-Lae Park

Soft Pneumatic Actuators (SPAs) have recently become popular for use as fingers in robotic hands because of their inherent compliance, low cost, and ease of construction. We seek to overcome two key limitations which limit SPAs' abilities to grasp and manipulate objects: 1) Current SPAs lack position or force sensor feedback, which prevents controlling them precisely (e. g. to achieve a hand preshape or apply a specified pushing force), and 2) the tip of the SPA is compliant and has high friction against common surfaces, causing the SPA to stick against surfaces when grasping objects from above. To overcome the first limitation we propose methods to integrate soft eGaIn sensors into SPAs and controllers that use these sensors' feedback for position and force control. To overcome the second limitation, we explore embedding rigid fingernails into the tip of the SPA so that the finger does not stick against surfaces and can wedge under objects. Our experiments suggest that we can achieve low steady-state error and overshoot in position and force using feed-forward models that relate pressure, force, and curvature along with a PID controller. We also compare several fingernail designs and show that the best-performing design significantly outperforms having no fingernails when grasping a set of common objects from a table.

IROS Conference 2015 Conference Paper

A framework for unsupervised online human reaching motion recognition and early prediction

Ruikun Luo
Dmitry Berenson

This paper focuses on recognition and prediction of human reaching motion in industrial manipulation tasks. Several supervised learning methods have been proposed for this purpose, but we seek a method that can build models on-the-fly and adapt to new people and new motion styles as they emerge. Thus, unlike previous work, we propose an unsupervised online learning approach to the problem, which requires no offline training or manual categorization of trajectories. Our approach consists of a two-layer library of Gaussian Mixture Models that can be used both for recognition and prediction. We do not assume that the number of motion classes is known a priori, and thus the library grows if it cannot explain a new observed trajectory. Given an observed portion of a trajectory, the framework can predict the remainder of the trajectory by first determining what GMM it belongs to, and then using Gaussian Mixture Regression to predict the remainder of the trajectory. We tested our method on motion-capture data recorded during assembly tasks. Our results suggest that the proposed framework outperforms supervised methods in terms of both recognition and prediction. We also show the benefit of using our two-layer framework over simpler approaches.

ICRA Conference 2015 Conference Paper

An online method for tight-tolerance insertion tasks for string and rope

Weifu Wang 0001
Dmitry Berenson
Devin J. Balkcom

This paper presents a fast tight-tolerance threading technique for string and rope. Instead of relying on simulations of these deformable objects to plan a path or compute control actions, we control the movement of the string with a virtual magnetic vector field emanating from the narrow openings we wish to thread through. We compute an approximate Jacobian to move the tip of the string through the vector field and propose a method to promote alignment of the head of the string to the opening. We also propose a method for re-grasping the string based on the relationship between the string's configuration, the orientation of the opening, and direction of gravity. This re-grasping method in conjunction with our controller can be used to thread the string through a sequence of openings. We evaluated our method in simulation (with simulated sensor noise) and on the Da Vinci surgical robot. Our results suggest that our method is quite robust to errors in sensing, and is capable of real-world threading tasks with the da Vinci robot, where the diameter of the string (3. 5mm) and opening (4. 9mm) differ by only 1. 4 mm.

ICRA Conference 2015 Conference Paper

Predicting human reaching motion in collaborative tasks using Inverse Optimal Control and iterative re-planning

Jim Mainprice
Rafi Hayne
Dmitry Berenson

To enable safe and efficient human-robot collaboration in shared workspaces, it is important for the robot to predict how a human will move when performing a task. While predicting human motion for tasks not known a priori is very challenging, we argue that single-arm reaching motions for known tasks in collaborative settings (which are especially relevant for manufacturing) are indeed predictable. Two hypotheses underlie our approach for predicting such motions: First, that the trajectory the human performs is optimal with respect to an unknown cost function, and second, that human adaptation to their partner's motion can be captured well through iterative replanning with the above cost function. The key to our approach is thus to learn a cost function which “explains” the motion of the human. To do this, we gather example trajectories from two participants performing a collaborative assembly task using motion capture. We then use Inverse Optimal Control to learn a cost function from these trajectories. Finally, we predict a human's motion for a given task by iteratively replanning a trajectory for a 23 DoF human kinematic model using the STOMP algorithm with the learned cost function in the presence of a moving collaborator. Our results suggest that our method outperforms baseline methods and generalizes well for tasks similar to those that were demonstrated.

ICRA Conference 2014 Conference Paper

A representation of deformable objects for motion planning with no physical simulation

Calder Phillips-Grafflin
Dmitry Berenson

We propose a new method of representing de-formable objects that allows both physical and qualitative properties to be captured in an efficient representation. We show how to use this representation with two types of motion planners: 1) optimal discrete planners, which are suitable for low-dimensional problems, 2) sampling-based planners that plan in high-dimensional cost spaces. In both cases, our representation allows us to formulate a cost function that directly assesses the cost of deformation without expensive physical simulation or computation of deformed geometry. We show that our methods can generate paths that minimize deformation in both simulated and physical environments with either hard and soft robots in either hard and soft environments. The efficiency of our representation allows these paths to be computed in under 20s for 3-DOF problems. For more complicated 6-DOF problems, low-deformation paths can be computed in under 120s. Additionally, using feedback from simulated and physical test environments, we demonstrate methods for calibrating models based on our representation.

ICRA Conference 2014 Conference Paper

DARPA Robotics Challenge: Towards a user-guided manipulation framework for high-DOF robots

Nicholas Alunni
Halit Bener Suay
Calder Phillips-Grafflin
Jim Mainprice
Dmitry Berenson
Sonia Chernova
Robert W. Lindeman
Daniel M. Lofaro

Supervision and teleoperation of high degree-of-freedom robots is a complex task due to environmental constraints such as obstacles and limited communication, as well as task specific requirements such as using more than one end-effector at the same time. In this work we present a supervision and teleoperation framework that allows an operator to see the surroundings of a robot in 3D, make necessary adjustments for a dual or single arm manipulation task, preview the task in simulation before execution, and finally execute the task on a real robot. The framework has been applied to the valve turning task of the DARPA Robotics Challenge on the PR2, Hubo2+, and DRCHubo robots.

IROS Conference 2014 Conference Paper

From autonomy to cooperative traded control of humanoid manipulation tasks with unreliable communication: System design and lessons learned

Jim Mainprice
Calder Phillips-Grafflin
Halit Bener Suay
Nicholas Alunni
Daniel M. Lofaro
Dmitry Berenson
Sonia Chernova
Robert W. Lindeman

In this paper, we report lessons learned through the design of a framework for teleoperating a humanoid robot to perform a manipulation task. We present a software framework for cooperative traded control that enables a team of operators to control a remote humanoid robot over an unreliable communications link. The framework produces statically-stable motion trajectories that are collision-free and respect end-effector pose constraints. After operator confirmation, these trajectories are sent over the data link for execution on the robot. Additionally, we have defined a clear operational procedure for the operators to manage the teleoperation task. We applied our system to the valve turning task in the DARPA Robotics Challenge (DRC). Our framework is able to perform reliably and is resilient to unreliable network conditions, as we demonstrate in a set of test runs performed remotely over the internet. We analyze our approach and discuss lessons learned which may be useful for others when designing such a system.

IROS Conference 2013 Conference Paper

Human-robot collaborative manipulation planning using early prediction of human motion

Jim Mainprice
Dmitry Berenson

In this paper we present a framework that allows a human and a robot to perform simultaneous manipulation tasks safely in close proximity. The proposed framework is based on early prediction of the human's motion. The prediction system, which builds on previous work in the area of gesture recognition, generates a prediction of human workspace occupancy by computing the swept volume of learned human motion trajectories. The motion planner then plans robot trajectories that minimize a penetration cost in the human workspace occupancy while interleaving planning and execution. Multiple plans are computed in parallel, one for each robot task available at the current time, and the trajectory with the least cost is selected for execution. We test our framework in simulation using recorded human motions and a simulated PR2 robot. Our results show that our framework enables the robot to avoid the human while still accomplishing the robot's task, even in cases where the initial prediction of the human's motion is incorrect. We also show that taking into account the predicted human workspace occupancy in the robot's motion planner leads to safer and more efficient interactions between the user and the robot than only considering the human's current configuration.

IROS Conference 2013 Conference Paper

Manipulation of deformable objects without modeling and simulating deformation

Dmitry Berenson

We present a method to manipulate deformable objects that does not require modeling and simulating deformation. Our method is based on the concept of diminishing rigidity, which we use to quickly compute an approximation to the Jacobian of the deformable object. This Jacobian is used to drive the points within the deformable object towards a set of targets. However, this Jacobian alone is insufficient to avoid stretching the object beyond its allowed length and to avoid gripper collision with obstacles. Thus a key part of our approach is incorporating techniques to avoid collision and excessive stretching. Our experiments show how to perform several interesting tasks for one and two-dimensional deformable objects using our method. They also show how the method can be applied to collaborative tasks, where the robot and a user simultaneously manipulate the deformable object. Our experiments are conducted in simulation but we emphasize that our method does not have access to the model of the deformable object used by the simulator, although we assume we are able to sense the geometry of the object. While our method is local, we find that it is quite versatile in the range of tasks it can perform, especially since it has no knowledge of the model of the deformable object.

ICRA Conference 2012 Conference Paper

A robot path planning framework that learns from experience

Dmitry Berenson
Pieter Abbeel
Ken Goldberg

We propose a framework, called Lightning, for planning paths in high-dimensional spaces that is able to learn from experience, with the aim of reducing computation time. This framework is intended for manipulation tasks that arise in applications ranging from domestic assistance to robot-assisted surgery. Our framework consists of two main modules, which run in parallel: a planning-from-scratch module, and a module that retrieves and repairs paths stored in a path library. After a path is generated for a new query, a library manager decides whether to store the path based on computation time and the generated path's similarity to the retrieved path. To retrieve an appropriate path from the library we use two heuristics that exploit two key aspects of the problem: (i) A correlation between the amount a path violates constraints and the amount of time needed to repair that path, and (ii) the implicit division of constraints into those that vary across environments in which the robot operates and those that do not. We evaluated an implementation of the framework on several tasks for the PR2 mobile manipulator and a minimally-invasive surgery robot in simulation. We found that the retrieve-and-repair module produced paths faster than planning-from-scratch in over 90% of test cases for the PR2 and in 58% of test cases for the minimally-invasive surgery robot.

ICRA Conference 2012 Conference Paper

Constellation - An algorithm for finding robot configurations that satisfy multiple constraints

Peter Kaiser 0001
Dmitry Berenson
Nikolaus Vahrenkamp
Tamim Asfour
Rüdiger Dillmann
Siddhartha S. Srinivasa

Planning motion for humanoid robots requires obeying simultaneous constraints on balance, collision-avoidance, and end-effector pose, among others. Several algorithms are able to generate configurations that satisfy these constraints given a good initial guess, i. e. a configuration which is already close to satisfying the constraints. However, when selecting goals for a planner a close initial guess is rarely available. Methods that attempt to satisfy all constraints through direct projection from a distant initial guess often fail due to opposing gradients for the various constraints, joint-limits, or singularities. We approach the problem of generating a constrained goal by searching for a configuration in the intersection of all constraint manifolds in configuration space (C-space). Starting with an initial guess, our algorithm, Constellation, builds a graph in C-space whose nodes are configurations that satisfy one or more constraints and whose cycles determine where the algorithm explores next. We compare the performance of our approach to direct projection and a previously-proposed cyclic projection method on reaching tasks for a humanoid robot with 33 DOF. We find that Constellation performs the best in terms of the number of solved queries across a wide range of problem difficulty. However, this success comes at higher computational cost.

ICRA Conference 2012 Conference Paper

Toward cloud-based grasping with uncertainty in shape: Estimating lower bounds on achieving force closure with zero-slip push grasps

Ben Kehoe
Dmitry Berenson
Ken Goldberg

This paper explores how Cloud Computing can facilitate grasping with shape uncertainty. We consider the most common robot gripper: a pair of thin parallel jaws, and a class of objects that can be modeled as extruded polygons. We model a conservative class of push-grasps that can enhance object alignment. The grasp planning algorithm takes as input an approximate object outline and Gaussian uncertainty around each vertex and center of mass. We define a grasp quality metric based on a lower bound on the probability of achieving force closure. We present a highly-parallelizable algorithm to compute this metric using Monte Carlo sampling. The algorithm uses Coulomb frictional grasp mechanics and a fast geometric test for conservative conditions for force closure. We run the algorithm on a set of sample shapes and compare the grasps with those from a planner that does not model shape uncertainty. We report computation times with single and multi-core computers and sensitivity analysis on algorithm parameters. We also describe physical grasp experiments using the Willow Garage PR2 robot.

ICRA Conference 2011 Conference Paper

Addressing cost-space chasms in manipulation planning

Dmitry Berenson
Thierry Siméon
Siddhartha S. Srinivasa

Finding paths in high-dimensional spaces becomes difficult when we wish to optimize the cost of a path in addition to obeying feasibility constraints. Recently the T-RRT algorithm was presented as a method to plan in high-dimensional cost-spaces and it was shown to perform well across a variety of problems. However, since the T-RRT relies solely on sampling to explore the space, it has difficulty navigating cost-space chasms-narrow low-cost regions surrounded by increasing cost. Such chasms are particularly common in planning for manipulators because many useful cost functions induce narrow or lower-dimensional low-cost areas. This paper presents the GradienT-RRT algorithm, which combines the T-RRT with a local gradient method to bias the search toward lower-cost regions. GradienT-RRT is effective at navigating chasms because it explores low-cost regions that are too narrow to explore by sampling alone. We compare the performance of T-RRT and GradienT-RRT on planning problems involving cost functions defined in workspace, task space, and C-space. We find that GradienT-RRT outperforms T-RRT in terms of the cost of the final path while maintaining better or comparable computation time. We also find that the cost of paths generated by GradienT-RRT is far less sensitive to changes in a key parameter, making it easier to tune the algorithm. Finally, we conclude with a demonstration of GradienT-RRT on a planning-with-uncertainty task on the physical HERB robot.

IROS Conference 2010 Conference Paper

People helping robots helping people: Crowdsourcing for grasping novel objects

Alexander Sorokin
Dmitry Berenson
Siddhartha S. Srinivasa
Martial Hebert

For successful deployment, personal robots must adapt to ever-changing indoor environments. While dealing with novel objects is a largely unsolved challenge in AI, it is easy for people. In this paper we present a framework for robot supervision through Amazon Mechanical Turk. Unlike traditional models of teleoperation, people provide semantic information about the world and subjective judgements. The robot then autonomously utilizes the additional information to enhance its capabilities. The information can be collected on demand in large volumes and at low cost. We demonstrate our approach on the task of grasping unknown objects.

ICRA Conference 2010 Conference Paper

Probabilistically complete planning with end-effector pose constraints

Dmitry Berenson
Siddhartha S. Srinivasa

We present a proof for the probabilistic completeness of RRT-based algorithms when planning with constraints on end-effector pose. Pose constraints can induce lower-dimensional constraint manifolds in the configuration space of the robot, making rejection sampling techniques infeasible. RRT-based algorithms can overcome this problem by using the sample-project method: sampling coupled with a projection operator to move configuration space samples onto the constraint manifold. Until now it was not known whether the sample-project method produces adequate coverage of the constraint manifold to guarantee probabilistic completeness. The proof presented in this paper guarantees probabilistic completeness for a class of RRT-based algorithms given an appropriate projection operator. This proof is valid for constraint manifolds of any fixed dimensionality.

IROS Conference 2009 Conference Paper

Addressing pose uncertainty in manipulation planning using Task Space Regions

Dmitry Berenson
Siddhartha S. Srinivasa
James J. Kuffner

We present an efficient approach to generating paths for a robotic manipulator that are collision-free and guaranteed to meet task specifications despite pose uncertainty. We first describe how to use task space regions (TSRs) to specify grasping and object placement tasks for a manipulator. We then show how to modify a set of TSRs for a certain task to take into account pose uncertainty. A key advantage of this approach is that if the pose uncertainty is too great to accomplish a certain task, we can quickly reject that task without invoking a planner. If the task is not rejected we run the IKBiRRT planner, which trades-off exploring the robot's C-space with sampling from TSRs to compute a path. Finally, we show several examples of a 7-DOF WAM arm planning paths in a cluttered kitchen environment where the poses of all objects are uncertain.

IROS Conference 2009 Conference Paper

Humanoid motion planning for dual-arm manipulation and re-grasping tasks

Nikolaus Vahrenkamp
Dmitry Berenson
Tamim Asfour
James J. Kuffner
Rüdiger Dillmann

In this paper, we present efficient solutions for planning motions of dual-arm manipulation and re-grasping tasks. Motion planning for such tasks on humanoid robots with a high number of degrees of freedom (DoF) requires computationally efficient approaches to determine the robot's full joint configuration at a given grasping position, i. e. solving the Inverse Kinematics (IK) problem for one or both hands of the robot. In this context, we investigate solving the inverse kinematics problem and motion planning for dual-arm manipulation and re-grasping tasks by combining a gradient-descent approach in the robot's pre-computed reachability space with random sampling of free parameters. This strategy provides feasible IK solutions at a low computation cost without resorting to iterative methods which could be trapped by joint-limits. We apply this strategy to dual-arm motion planning tasks in which the robot is holding an object with one hand in order to generate whole-body robot configurations suitable for grasping the object with both hands. In addition, we present two probabilistically complete RRT-based motion planning algorithms (J+-RRT and IK-RRT) that interleave the search for an IK solution with the search for a collision-free trajectory and the extension of these planners to solving re-grasping problems. The capabilities of combining IK methods and planners are shown both in simulation and on the humanoid robot ARMAR-III performing dual-arm tasks in a kitchen environment.

ICRA Conference 2009 Conference Paper

Manipulation planning on constraint manifolds

Dmitry Berenson
Siddhartha S. Srinivasa
Dave Ferguson 0001
James J. Kuffner

We present the Constrained Bi-directional Rapidly-Exploring Random Tree (CBiRRT) algorithm for planning paths in configuration spaces with multiple constraints. This algorithm provides a general framework for handling a variety of constraints in manipulation planning including torque limits, constraints on the pose of an object held by a robot, and constraints for following workspace surfaces. CBiRRT extends the Bi-directional RRT (BiRRT) algorithm by using projection techniques to explore the configuration space manifolds that correspond to constraints and to find bridges between them. Consequently, CBiRRT can solve many problems that the BiRRT cannot, and only requires one additional parameter: the allowable error for meeting a constraint. We demonstrate the CBiRRT on a 7DOF WAM arm with a 4DOF Barrett hand on a mobile base. The planner allows this robot to perform household tasks, solve puzzles, and lift heavy objects.

ICRA Conference 2009 Conference Paper

Manipulation planning with Workspace Goal Regions

Dmitry Berenson
Siddhartha S. Srinivasa
Dave Ferguson 0001
Alvaro Collet
James J. Kuffner

We present an approach to path planning for manipulators that uses Workspace Goal Regions (WGRs) to specify goal end-effector poses. Instead of specifying a discrete set of goals in the manipulator's configuration space, we specify goals more intuitively as volumes in the manipulator's workspace. We show that WGRs provide a common framework for describing goal regions that are useful for grasping and manipulation. We also describe two randomized planning algorithms capable of planning with WGRs. The first is an extension of RRT-JT that interleaves exploration using a Rapidly-exploring Random Tree (RRT) with exploitation using Jacobian-based gradient descent toward WGR samples. The second is the IKBiRRT algorithm, which uses a forward-searching tree rooted at the start and a backward-searching tree that is seeded by WGR samples. We demonstrate both simulation and experimental results for a 7DOF WAM arm with a mobile base performing reaching and pick-and-place tasks. Our results show that planning with WGRs provides an intuitive and powerful method of specifying goals for a variety of tasks without sacrificing efficiency or desirable completeness properties.

ICRA Conference 2009 Conference Paper

Object recognition and full pose registration from a single image for robotic manipulation

Alvaro Collet
Dmitry Berenson
Siddhartha S. Srinivasa
Dave Ferguson 0001

Robust perception is a vital capability for robotic manipulation in unstructured scenes. In this context, full pose estimation of relevant objects in a scene is a critical step towards the introduction of robots into household environments. In this paper, we present an approach for building metric 3D models of objects using local descriptors from several images. Each model is optimized to fit a set of calibrated training images, thus obtaining the best possible alignment between the 3D model and the real object. Given a new test image, we match the local descriptors to our stored models online, using a novel combination of the RANSAC and Mean Shift algorithms to register multiple instances of each object. A robust initialization step allows for arbitrary rotation, translation and scaling of objects in the test images. The resulting system provides markerless 6-DOF pose estimation for complex objects in cluttered scenes. We provide experimental results demonstrating orientation and translation accuracy, as well a physical implementation of the pose output being used by an autonomous robot to perform grasping in highly cluttered scenes.

ICRA Conference 2008 Conference Paper

An optimization approach to planning for mobile manipulation

Dmitry Berenson
James J. Kuffner
Howie Choset

We present an optimization-based approach to grasping and path planning for mobile manipulators. We focus on pick-and-place operations, where a given object must be moved from its start configuration to its goal configuration by the robot. Given only the start and goal configurations of the object and a model of the robot and scene, our algorithm finds a grasp and a trajectory for the robot that will bring the object to its goal configuration. The algorithm consists of two phases: optimization and planning. In the optimization phase, the optimal robot configurations and grasp are found for the object in its start and goal configurations using a co-evolutionary algorithm. In the planning phase, a path is found connecting the two robot configurations found by the optimization phase using Rapidly-Exploring Random Trees (RRTs). We benchmark our algorithm and demonstrate it on a 10 DOF mobile manipulator performing complex pick-and-place tasks in simulation.