Ken Goldberg

ICRA Conference 2025 Conference Paper

Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset

• Andrew Goldberg 0001
• Kavish Kondap
• Tianshuang Qiu
• Zehan Ma
• Letian Fu
• Justin Kerr
• Huang Huang
• Kaiyuan Chen 0001

Generative AI systems have shown impressive capabilities in creating text, code, and images. Inspired by the importance of research in industrial Design for Assembly, we introduce a novel problem: Generative Design-for-RobotAssembly (GDfRA). The task is to generate an assembly based on a natural language prompt (e. g. , “giraffe”) and an image of available physical components, such as 3D-printed blocks. The output is an assembly, a spatial arrangement of these components, accompanied by instructions for a robot to build it. The output geometry must 1) resemble the requested object and 2) be reliably assembled by a 6 DoF robot arm with a suction gripper. We then present Blox-Net, a GDfRA system that combines generative vision language models with well-established methods in computer vision, simulation, perturbation analysis, motion planning, and physical robot experimentation to solve a class of GDfRA problems without human supervision. Blox-Net achieved a Top-1 accuracy of $\mathbf{6 3. 5 \%}$ in the semantic accuracy of its designed assemblies. Six designs, after Blox-Net's automated pertubation redesign, were reliably assembled by a robot, achieving near-perfect success across $\mathbf{1 0}$ consecutive assembly iterations with human intervention only during reset prior to assembly. The entire pipeline from the textual word to reliable physical assembly is performed without human intervention. Project Page: https://bloxnet.org/

IROS Conference 2025 Conference Paper

Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats

• Simeon Adebola
• Chung Min Kim
• Justin Kerr
• Shuangyu Xie
• Prithvi Akella
• Jose Luis Susa Rincon
• Eugen Solowjow
• Ken Goldberg

Commercial plant phenotyping systems using fixed cameras cannot perceive many plant details due to leaf occlusion. In this paper, we present Botany-Bot, a system for building detailed “annotated digital twins” of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial robot arm, and 3D segmentated Gaussian Splat models. We also present robot algorithms for manipulating leaves to take high-resolution indexable images of occluded details such as stem buds and the underside/topside of leaves. Results from experiments suggest that Botany-Bot can segment leaves with 90. 8% accuracy, detect leaves with 86. 2% accuracy, lift/push leaves with 77. 9% accuracy, and take detailed overside/underside images with 77. 3% accuracy. Code, videos, and datasets are available at https://berkeleyautomation.github.io/Botany-Bot/.

ICRA Conference 2025 Conference Paper

Energy Efficient Planning for Repetitive Heterogeneous Tasks in Precision Agriculture

• Shuangyu Xie
• Ken Goldberg
• Dezhen Song

Robotic weed removal in precision agriculture introduces a repetitive heterogeneous task planning (RHTP) challenge for a mobile manipulator. RHTP has two unique characteristics: 1) an observe-first-and-manipulate-later (OFML) temporal constraint that forces a unique ordering of two different tasks for each target and 2) energy savings from efficient task collocation to minimize unnecessary movements. RHTP can be framed as a stochastic renewal process. According to the Renewal Reward Theorem, the expected energy usage per task cycle is the long-run average. Traditional task and motion planning focuses on feasibility rather than optimality due to the unknown object and obstacle position prior to execution. However, the known target/obstacle distribution in precision agriculture allows minimizing the expected energy usage. For each instance in this renewal process, we first compute task space partition, a novel data structure that computes all possibilities of task multiplexing and its probabilities with robot reachability. Then we propose a region-based setcoverage problem to formulate the RHTP as a mixed-integer nonlinear programming. We have implemented and solved RHTP using Branch-and-Bound solver. Compared to a baseline in simulations based on real field data, the results suggest a significant improvement in path length, number of robot stops, overall energy usage, and number of replans.

ICRA Conference 2025 Conference Paper

FogROS2-PLR: Probabilistic Latency-Reliability for Cloud Robotics

• Kaiyuan Chen 0001
• Nan Tian
• Christian Juette
• Tianshuang Qiu
• Liu Ren
• John Kubiatowicz
• Ken Goldberg

Cloud robotics enables robots to offload computationally intensive tasks to cloud servers for performance, cost, and ease of management. However, the network and cloud computing infrastructure are not designed for reliable timing guarantees, due to fluctuating Quality-of-Service (QoS). In this work, we formulate an impossibility triangle theorem for: Latency reliability, Singleton server, and Commodity hardware. The LSC theorem suggests that providing replicated servers with uncorrelated failures can exponentially reduce the probability of missing a deadline. We present FogROS2-Probabilistic Latency Reliability (PLR) that uses multiple independent network interfaces to send requests to replicated cloud servers and uses the first response back. We design routing mechanisms to discover, connect, and route through non-default network interfaces on robots. FogROS2-PLR optimizes the selection of interfaces to servers to minimize the probability of missing a deadline. We conduct a cloud-connected driving experiment with two 5 G service providers, demonstrating FogROS2-PLR effectively provides smooth service quality even if one of the service providers experiences low coverage and base station handover. We use 99 Percentile (P99) latency to evaluate anomalous long-tail latency behavior. In one experiment, FogROS2-PLR improves P99 latency by up to 3. 7 x compared to using one service provider. We deploy FogROS2-PLR on a physical Stretch 3 robot performing an indoor human-tracking task. Even in a fully covered $\text{Wi}-\text{Fi}$ and 5 G environment, FogROS2-PLR improves the responsiveness of the robot reducing mean latency by 36% and P99 latency by 33%. Code and supplementary can be found on website 1 1 https://github.com/data-capsule/rt-fogros2.

ICRA Conference 2025 Conference Paper

ICRT: In-Context Imitation Learning via Next-Token Prediction

• Max Fu
• Huang Huang
• Gaurav Datta
• Lawrence Yunliang Chen
• William Chung-Ho Panitch
• Fangchen Liu
• Hui Li
• Ken Goldberg

In-context imitation learning is the capability to perform novel tasks when prompted with task demonstration examples. In-Context Robot Transformer (ICRT) is a causal transformer that performs autoregressive prediction on sensorimotor trajectories, which include images, proprioceptive states, and actions. This approach supports flexible and training-free execution of new tasks at test time. Experiments with a Franka Emika robot demonstrate that ICRT can adapt to new environment configurations that differ from both the prompt and the training data. In a multi-task environment setup, ICRT significantly outperforms current state-of-the-art robot foundation models on generalization to unseen tasks. Code, data, and appendix are available on https://icrt.dev.

IROS Conference 2025 Conference Paper

Omni-Scan: Creating Visually-Accurate Digital Twin Object Models Using a Bimanual Robot with Handover and Gaussian Splat Merging

• Tianshuang Qiu
• Zehan Ma
• Karim El-Refai
• Hiya Shah
• Chung Min Kim
• Justin Kerr
• Ken Goldberg

3D Gaussian Splats (3DGSs) are 3D object models derived from multi-view images. Such “digital twins” are useful for simulations, virtual reality, E-commerce, robot policy fine-tuning, and part inspection. 3D object scanning usually requires multi-camera arrays, precise laser scanners, or robot wrist-mounted cameras, which have restricted workspaces. We propose Omni-Scan, a pipeline for producing high-quality 3D Gaussian Splat models using a bi-manual robot that grasps an object with one gripper and rotates the object with respect to one stationary camera. The object is then re-grasped by a second gripper to expose surfaces that were occluded by the first gripper. We present the Omni-Scan robot pipeline using DepthAnything, Segment Anything, as well as RAFT optical flow models to identify and isolate objects held by a robot gripper while removing the gripper and the background. We then modify the 3DGS training pipeline to support concatenated datasets with gripper occlusion, producing an omni-directional (360°) model of the object. We apply Omni-Scan to part defect inspection, finding that it can identify visual or geometric defects in 12 different industrial and household objects with an average accuracy of 83. 3%. More details and interactive videos of Omni-Scan 3DGS models can be found at https://berkeleyautomation.github.io/omni-scan/.

ICML Conference 2025 Conference Paper

OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction

• Huang Huang
• Fangchen Liu
• Letian Fu
• Tingfan Wu
• Mustafa Mukadam
• Jitendra Malik
• Ken Goldberg
• Pieter Abbeel

Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions. Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies, degrading the pre-trained semantic alignments. We propose OTTER, a novel VLA architecture that leverages these existing alignments through explicit, text-aware visual feature extraction. Instead of processing all visual features, OTTER selectively extracts and passes only task-relevant visual features that are semantically aligned with the language instruction to the policy transformer. This allows OTTER to keep the pre-trained vision-language encoders frozen. Thereby, OTTER preserves and utilizes the rich semantic understanding learned from large-scale pre-training, enabling strong zero-shot generalization capabilities. In simulation and real-world experiments, OTTER significantly outperforms existing VLA models, demonstrating strong zero-shot generalization to novel objects and environments. Video, code, checkpoints, and dataset: https: //ottervla. github. io/.

ICRA Conference 2025 Conference Paper

Persistent Object Gaussian Splat (POGS) for Tracking Human and Robot Manipulation of Irregularly Shaped Objects

• Justin Yu
• Kush Hari
• Karim El-Refai
• Arnav Dalal
• Justin Kerr
• Chung Min Kim
• Richard Cheng
• Muhammad Zubair Irshad

Tracking and manipulating irregularly-shaped, previously unseen objects in dynamic environments is important for robotic applications in manufacturing, assembly, and logistics. Recently introduced Gaussian Splats [1] efficiently model object geometry, but lack persistent state estimation for taskoriented manipulation. We present Persistent Object Gaussian Splat (POGS), a system that embeds semantics, self-supervised visual features, and object grouping features into a compact representation that can be continuously updated to estimate the pose of scanned objects. POGS updates object states without requiring expensive rescanning or prior CAD models of objects. After an initial multi-view scene capture and training phase, POGS uses a single stereo camera to integrate depth estimates along with self-supervised vision encoder features for object pose estimation. POGS supports grasping, reorientation, and natural language-driven manipulation by refining object pose estimates, facilitating sequential object reset operations with human-induced object perturbations and tool servoing, where robots recover tool pose despite tool perturbations of up to 30°. POGS achieves up to 12 consecutive successful object resets and recovers from 80% of in-grasp tool perturbations.

IROS Conference 2025 Conference Paper

PyRoki: A Modular Toolkit for Robot Kinematic Optimization

• Chung Min Kim
• Brent Yi
• Hongsuk Choi
• Yi Ma 0001
• Ken Goldberg
• Angjoo Kanazawa

Robot motion can have many goals. Depending on the task, we might optimize for pose error, speed, collision, or similarity to a human demonstration. Motivated by this, we present PyRoki: a modular, extensible, and deviceagnostic toolkit for solving kinematic optimization problems. PyRoki couples an interface for specifying kinematic variables and costs with an efficient nonlinear least squares optimizer. Unlike existing tools, it is also device-agnostic: optimization runs natively on CPU, GPU, and TPU. In this paper, we present (i) the design and implementation of PyRoki, (ii) motion retargeting and planning case studies that highlight the advantages of PyRoki’s modularity, and (iii) optimization benchmarking, where PyRoki can be 1. 4-1. 7x faster and converges to lower errors than cuRobo, an existing GPU-accelerated inverse kinematics library. The code is open-sourced at https://pyroki-toolkit.github.io.

TMLR Journal 2025 Journal Article

Rethinking Patch Dependence for Masked Autoencoders

• Letian Fu
• Long Lian
• Renhao Wang
• Baifeng Shi
• Xudong Wang
• Adam Yala
• Trevor Darrell
• Alexei A Efros

In this work, we examine the impact of inter-patch dependencies in the decoder of masked autoencoders (MAE) on representation learning. We decompose the decoding mechanism for masked reconstruction into self-attention between mask tokens and cross-attention between masked and visible tokens. Our findings reveal that MAE reconstructs coherent images from visible patches not through interactions between patches in the decoder but by learning a global representation within the encoder. This discovery leads us to propose a simple visual pretraining framework: cross-attention masked autoencoders (CrossMAE). This framework employs only cross-attention in the decoder to independently read out reconstructions for a small subset of masked patches from encoder outputs. This approach achieves comparable or superior performance to traditional MAE across models ranging from ViT-S to ViT-H and significantly reduces computational requirements. By its design, CrossMAE challenges the necessity of interaction between mask tokens for effective masked pretraining. Code and models are publicly available: https://crossmae.github.io/

ICRA Conference 2025 Conference Paper

Robo-DM: Data Management for Large Robot Datasets

• Kaiyuan Chen 0001
• Letian Fu
• David Huang
• Yanxiang Zhang
• Lawrence Yunliang Chen
• Huang Huang
• Kush Hari
• Ashwin Balakrishna

Recent results suggest that very large datasets of teleoperated robot demonstrations can be used to train transformer-based models that have the potential to generalize to new scenes, robots, and tasks. However, curating, distributing, and loading large datasets of robot trajectories, which typically consist of video, textual, and numerical modalities - including streams from multiple cameras - remains challenging. We propose Robo-DM, an efficient open-source cloud-based data management toolkit for collecting, sharing, and learning with robot data. With Robo-DM, robot datasets are stored in a self-contained format with Extensible Binary Meta Language (EBML). Robo-DM can significantly reduce the size of robot trajectory data, transfer costs, and data load time during training. Compared to the RLDS format used in OXE datasets, Robo-DM's compression saves space by up to 70x (lossy) and 3. 5x (lossless). Robo-DM also accelerates data retrieval by load-balancing video decoding with memory-mapped decoding caches. Compared to LeRobot, a framework that also uses lossy video compression, Robo-DM is up to 50x faster when decoding sequentially. We physically evaluate a model trained by Robo-DM with lossy compression, a pick-and-place task, and In-Context Robot Transformer. Robo-DM uses 75x compression of the original dataset and does not suffer reduction in downstream task accuracy. Code and evaluation scripts can be found on website https://github.com/BerkeleyAutomation/fog_x.

NeurIPS Conference 2025 Conference Paper

Robo2VLM: Improving Visual Question Answering using Large-Scale Robot Manipulation Data

• Kaiyuan Eric Chen
• Shuangyu Xie
• Zehan Ma
• Pannag Sanketi
• Ken Goldberg

Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora. They can augment robotic systems with scene understanding and task planning, and assist visuomotor policies that are trained on robot trajectory data. We explore the reverse paradigm — using rich, real, multi-modal robot trajectory data to enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs. Given a human tele-operated robot demonstration with video and robot data, Robo2VLM derives ground-truth from non-visual and non-descriptive sensory modalities, such as end-effector pose, gripper aperture, and force sensing. Based on these modalities, it segments the robot trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses scene and interaction understanding to identify 3D properties of the robot, task goal, and the target object. The properties are used to generate representative VQA queries – images with textural multiple-choice questions – based on spatial, goal-conditioned, and interaction reasoning question templates. We use a subset of Open X-Embodiment to generate Robo2VLM-1, a large-scale in-the-wild dataset with 684, 710 questions based on 463 distinct scenes and 3, 396 robotic manipulation tasks from 176k real robot trajectories. Results suggest that Robo2VLM-1 can benchmark and improve VLM capabilities in spatial and interaction reasoning.

ICRA Conference 2025 Conference Paper

SuFIA-BC: Generating High Quality Demonstration Data for Visuomotor Policy Learning in Surgical Subtasks

• Masoud Moghani
• Nigel Nelson
• Mohamed Ghanem
• Andres Diaz-Pinto
• Kush Hari
• Mahdi Azizian
• Ken Goldberg
• Sean Huver

Behavior cloning facilitates the learning of dexterous manipulation skills, yet the complexity of surgical environments, the difficulty and expense of obtaining patient data, and robot calibration errors present unique challenges for surgical robot learning. We provide an enhanced surgical digital twin with photorealistic human anatomical organs, integrated into a comprehensive simulator designed to generate high-quality synthetic data to solve fundamental tasks in surgical autonomy. We present SuFIA-BC: visual Behavior Cloning policies for Surgical First Interactive Autonomy Assistants. We investigate visual observation spaces including multi-view cameras and 3D visual representations extracted from a single endoscopic camera view. Through systematic evaluation, we find that the diverse set of photorealistic surgical tasks introduced in this work enables a comprehensive evaluation of prospective behavior cloning models for the unique challenges posed by surgical environments. We observe that current state-of-the-art behavior cloning techniques struggle to solve the contact-rich and complex tasks evaluated in this work, regardless of their underlying perception or control architectures. These findings highlight the importance of customizing perception pipelines and control architectures, as well as curating larger-scale synthetic datasets that meet the specific demands of surgical tasks. Project website: orbit-surgical.github.io/sufia-bc/

IROS Conference 2025 Conference Paper

Surgical D-Knot: Augmented Dexterity for Tying Double Knots by Monitoring Optical Flow in Monocular Attention Windows

• Ziyang Chen
• Kush Hari
• Tanmayi Dasari
• Karen Shieh
• Ria Jain
• Danyal M. Fer
• Gary Guthart
• Ken Goldberg

Knot tying is a fundamental dexterous surgical subtask that is a key step in suturing. One challenge to robot augmentation is limited depth perception due to the small baseline of surgical endoscopic cameras. In this work, we present Surgical D-Knot: an augmented dexterity pipeline combining learned perception with model-based methods to perform surgical double knots using only one monocular RGB camera. This pipeline includes 2D grasp point identification, 3D suture thread grasping using local feature servoing, suture thread wrapping using relative motion and 2D re-grasp point identification. Human dexterity is required for initial thread setup and thread cutting after each double knot. Physical experiments with 120 double knot trials result in a success rate of 80. 83% for the initial knot and 55. 83% for the second knot. Translation of surgical knot tying to chicken skin results in success rates of 73. 75% for the initial knot and 40% for the second knot. Each double knot requires on average 70 seconds. This is the first work to our knowledge that augments human dexterity for double knot tying. https://sites.google.com/view/surgicaldknot/

ICML Conference 2024 Conference Paper

A Touch, Vision, and Language Dataset for Multimodal Alignment

• Letian Fu
• Gaurav Datta
• Huang Huang
• William Chung-Ho Panitch
• Jaimyn Drake
• Joseph Ortiz
• Mustafa Mukadam
• Mike Lambeta

Touch is an important sensing modality for humans, but it has not yet been incorporated into a multimodal generative language model. This is partially due to the difficulty of obtaining natural language labels for tactile data and the complexity of aligning tactile readings with both visual observations and language descriptions. As a step towards bridging that gap, this work introduces a new dataset of 44K in-the-wild visiontouch pairs, with English language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V (90%). We use this dataset to train a vision-language-aligned tactile encoder for open-vocabulary classification and a touch-visionlanguage (TVL) model for text generation using the trained encoder. Results suggest that by incorporating touch, the TVL model improves (+29% classification accuracy) tactile-vision-language alignment over existing models trained on any pair of those modalities. Although only a small fraction of the dataset is human labeled, the TVL model demonstrates improved visual-tactile understanding over GPT-4V (+12%) and open-source vision-language models (+32%) on a new touch-vision understanding benchmark. Code, checkpoints and data are available on https: //tactile-vlm. github. io.

IROS Conference 2024 Conference Paper

BOMP: Bin-Optimized Motion Planning

• Zachary Tam
• Karthik Dharmarajan
• Tianshuang Qiu
• Yahav Avigal
• Jeffrey Ichnowski
• Ken Goldberg

In logistics, the ability to quickly compute and execute pick-and-place motions from bins is critical to increasing productivity. We present Bin-Optimized Motion Planning (BOMP), a motion planning framework that plans arm motions for a six-axis industrial robot with a long-nosed suction tool to remove boxes from deep bins. BOMP considers robot arm kinematics, actuation limits, the dimensions of a grasped box, and a varying height map of a bin environment to rapidly generate time-optimized, jerk-limited, and collision-free trajectories. The optimization is warm-started using a deep neural network trained offline in simulation with 25, 000 scenes and corresponding trajectories. Experiments with 96 simulated and 15 physical environments suggest that BOMP generates collision-free trajectories that are up to 58% faster than baseline sampling-based planners and up to 36% faster than an industry-standard Up-Over-Down algorithm, which has an extremely low 15% success rate in this context. BOMP also generates jerk-limited trajectories while baselines do not. Website: https://sites.google.com/berkeley.edu/bomp.

ICRA Conference 2024 Conference Paper

Conformal Policy Learning for Sensorimotor Control under Distribution Shifts

• Huang Huang
• Satvik Sharma
• Antonio Loquercio
• Anastasios N. Angelopoulos
• Ken Goldberg
• Jitendra Malik

This paper focuses on the problem of detecting and reacting to changes in the distribution of a sensorimotor controller’s observables. The key idea is the design of policies that can take conformal quantiles as input, to detect distribution shifts with formal statistical guarantees, which we define as conformal policy learning. We show how to design such policies by using conformal quantiles to switch between base policies with different characteristics, e. g. safety or speed, or directly augmenting a policy observation with a quantile and training it with reinforcement learning. Theoretically, we show that such policies achieve the formal convergence guarantees in finite time. In addition, we thoroughly evaluate their advantages and limitations on two use cases: simulated autonomous driving and active perception with a physical quadruped. Empirical results demonstrate that our approach outperforms five baselines. It is also the simplest of the baseline strategies besides one ablation. Being easy to use, flexible, and with formal guarantees, our work demonstrates how conformal prediction can be an effective tool for sensorimotor learning under uncertainty.

ICRA Conference 2024 Conference Paper

FogROS2-Config: A Toolkit for Choosing Server Configurations for Cloud Robotics

• Kaiyuan Chen 0001
• Kush Hari
• Rohil Khare
• Charlotte Le
• Trinity Chung
• Jaimyn Drake
• Jeffrey Ichnowski
• John Kubiatowicz

Cloud service providers provide over 50, 000 distinct and dynamically changing set of cloud server options. To help roboticists make cost-effective decisions, we present FogROS2-Config, an open toolkit that takes ROS2 nodes as input and automatically runs relevant benchmarks to quickly return a menu of cloud compute services that tradeoff latency and cost. Because it is infeasible to try every hardware configuration, FogROS2-Config quickly samples tests a small set of edge-case servers. We evaluate FogROS2-Config on three robotics application tasks: visual SLAM, grasp planning. and motion planning. FogROS2-Config can reduce the cost by up to 20x. By comparing with a Pareto frontier for cost and latency by running the application task on feasible server configurations, we evaluate cost and latency models and confirm that FogROS2-Config selects efficient hardware configurations to balance cost and latency. Videos and code are available on the website https://sites.google.com/view/fogros2-config

IROS Conference 2024 Conference Paper

FogROS2-FT: Fault Tolerant Cloud Robotics

• Kaiyuan Chen 0001
• Kush Hari
• Trinity Chung
• Michael Wang
• Nan Tian
• Christian Juette
• Jeffrey Ichnowski
• Liu Ren

Cloud robotics enables robots to offload complex computational tasks to cloud servers for performance and ease of management. However, cloud compute can be costly, cloud services can suffer occasional downtime, and connectivity between the robot and cloud can be prone to variations in network Quality-of-Service (QoS). We present FogROS2-FT (Fault Tolerant) to mitigate these issues by introducing a multi-cloud extension that automatically replicates independent stateless robotic services, routes requests to these replicas, and directs the first response back. With replication, robots can still benefit from cloud computations even when a cloud service provider is down or there is low QoS. Additionally, many cloud computing providers offer low-cost "spot" computing instances that may shutdown unpredictably. Normally, these low-cost instances would be inappropriate for cloud robotics, but the fault tolerance nature of FogROS2-FT allows them to be used reliably. We demonstrate FogROS2-FT fault tolerance capabilities in 3 cloud-robotics scenarios in simulation (visual object detection, semantic segmentation, motion planning) and 1 physical robot experiment (scan-pick-and-place). Running on the same hardware specification, FogROS2-FT achieves motion planning with up to 2. 2x cost reduction and up to a 5. 53x reduction on 99 Percentile (P99) long-tail latency. FogROS2-FT reduces the P99 long-tail latency of object detection and semantic segmentation by 2. 0x and 2. 1x, respectively, under network slowdown and resource contention. Videos and code are available at https://sites.google.com/view/fogros2-ft.

ICRA Conference 2024 Conference Paper

FogROS2-LS: A Location-Independent Fog Robotics Framework for Latency Sensitive ROS2 Applications

• Kaiyuan Chen 0001
• Michael Wang
• Marcus Gualtieri
• Nan Tian
• Christian Juette
• Liu Ren
• Jeffrey Ichnowski
• John Kubiatowicz

In Cloud Robotics, long system latency due to varying network conditions can cause instability and collisions. However, this can be minimized in the almost univeral case where there are multiple sources available for cloud servers. By extending anycast routing, we introduce FogROS2-Latency-Sensitive, a Fog Robotics framework that offers secure, location-independent connections between robots and latency-sensitive cloud-based servers. FogROS2-LS offloads conventional on-board state estimators and feedback controllers to Cloud and Edge compute hardware without modifying existing applications in ROS2. In the presence of multiple identical services, FogROS2-LS dynamically identifies and transitions to the optimal service deployment that meets latency requirements, thereby empowering robots with limited on-board computing capacity to safely and efficiently navigate dynamic, human-dense environments. We evaluate FogROS2-LS with two latency sensitive case studies: (1) Collision Avoidance: a robot arm guided by visual feedback from consistent distance estimation and collision checking on Cloud and Edge. FogROS2-LS reduces collision failures by up to 8. 5x by selecting the best available server, and (2) Target Tracking: FogROS2-LS enables robust and continuous target following and can recover from network failures. Videos and code are available on the website https://sites.google.com/view/fogros2-ls.

IROS Conference 2024 Conference Paper

IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learning

• Ryan Hoque
• Ajay Mandlekar
• Caelan Reed Garrett
• Ken Goldberg
• Dieter Fox

Imitation learning is a promising paradigm for training robot control policies, but these policies can suffer from distribution shift, where the conditions at evaluation time differ from those in the training data. A popular approach for increasing policy robustness to distribution shift is interactive imitation learning (i. e. , DAgger and variants), where a human operator provides corrective interventions during policy rollouts. However, collecting a sufficient amount of interventions to cover the distribution of policy mistakes can be burdensome for human operators. We propose IntervenGen (I-Gen), a novel data augmentation system for robot control that autonomously produces a large set of corrective interventions with rich coverage of the state space from a small number of human interventions. We apply I-Gen to 4 simulated environments and 1 physical environment with object pose estimation error and show that it can increase policy robustness by up to 39× with only 10 human interventions. Videos and more results are available at https://sites.google.com/view/intervengen2024.

IROS Conference 2024 Conference Paper

Language-Embedded Gaussian Splats (LEGS): Incrementally Building Room-Scale Representations with a Mobile Robot

• Justin Yu
• Kush Hari
• Kishore Srinivas
• Karim El-Refai
• Adam Rashid
• Chung Min Kim
• Justin Kerr
• Richard Cheng

Building semantic 3D maps is valuable for searching for objects of interest in offices, warehouses, stores, and homes. We present a mapping system that incrementally builds a Language-Embedded Gaussian Splat (LEGS): a detailed 3D scene representation that encodes both appearance and semantics in a unified representation. LEGS is trained online as a robot traverses its environment to enable localization of open-vocabulary object queries. We evaluate LEGS on 4 room-scale scenes where we query for objects in the scene to assess how LEGS can capture semantic meaning. We compare LEGS to LERF [1] and find that while both systems have comparable object query success rates, LEGS trains over 3. 5x faster than LERF. Results suggest that a multi-camera setup and incremental bundle adjustment can boost visual reconstruction quality in constrained robot trajectories, and suggest LEGS can localize open-vocabulary and long-tail object queries with up to 66% accuracy. See project website at: berkeleyautomation.github.io/LEGS

ICRA Conference 2024 Conference Paper

Lifelong LERF: Local 3D Semantic Inventory Monitoring Using FogROS2

• Adam Rashid
• Chung Min Kim
• Justin Kerr
• Letian Fu
• Kush Hari
• Ayah Ahmad
• Kaiyuan Chen 0001
• Huang Huang

Inventory monitoring in homes, factories, and retail stores relies on maintaining data despite objects being swapped, added, removed, or moved. We introduce Lifelong LERF, a method that allows a mobile robot with minimal compute to jointly optimize a dense language and geometric representation of its surroundings. Lifelong LERF maintains this representation over time by detecting semantic changes and selectively updating these regions of the environment, avoiding the need to exhaustively remap. Human users can query inventory by providing natural language queries and receiving a 3D heatmap of potential object locations. To manage the computational load, we use Fog-ROS2, a cloud robotics platform, to offload resource-intensive tasks. Lifelong LERF obtains poses from a monocular RGBD SLAM backend, and uses these poses to progressively optimize a Language Embedded Radiance Field (LERF) for semantic monitoring. Experiments with 3-5 objects arranged on a tabletop and a Turtlebot with a RealSense camera suggest that Lifelong LERF can persistently adapt to changes in objects with up to 91% accuracy.

IROS Conference 2024 Conference Paper

MANIP: A Modular Architecture for Integrating Interactive Perception for Robot Manipulation

• Justin Yu
• Tara Sadjadpour
• Abby O'Neill
• Mehdi Khfifi
• Lawrence Yunliang Chen
• Richard Cheng
• Muhammad Zubair Irshad
• Ashwin Balakrishna

We propose a modular systems architecture, MANIP, that can facilitate the design and development of robot manipulation systems by systematically combining learned subpolicies with well-established procedural algorithmic primitives such as Inverse Kinematics, Kalman Filters, RANSAC outlier rejection, PID modules, etc. (aka "Good Old Fashioned Engineering (GOFE)"). The MANIP architecture grew from our lab’s experience developing robot systems for folding clothes, routing cables, and untangling knots. To address failure modes, MANIP can facilitate inclusion of "interactive perception" subpolicies that execute robot actions to modify system state to bring the system into alignment with the training distribution and / or to disambiguate system state when system state confidence is low. We demonstrate how MANIP can be applied with 3 case studies and then describe a detailed case study in cable tracing with experiments that suggest MANIP can improve performance by up to 88%. Code and details are available at: https://berkeleyautomation.github.io/MANIP/

ICRA Conference 2024 Conference Paper

Manipulator as a Tail: Promoting Dynamic Stability for Legged Locomotion

• Huang Huang
• Antonio Loquercio
• Ashish Kumar 0007
• Neerja Thakkar
• Ken Goldberg
• Jitendra Malik

For locomotion, is an arm on a legged robot a liability or an asset for locomotion? Biological systems evolved additional limbs beyond legs that facilitates postural control. This work shows how a manipulator can be an asset for legged locomotion at high speeds or under external perturbations, where the arm serves beyond manipulation. Since the system has 15 degrees of freedom (twelve for the legged robot and three for the arm), off-the-shelf reinforcement learning (RL) algorithms struggle to learn effective locomotion policies. Inspired by Bernstein’s neurophysiological theory of animal motor learning, we develop an incremental training procedure that initially freezes some degrees of freedom and gradually releases them, using behaviour cloning (BC) from an early learning procedure to guide optimization in later learning. Simulation experiments show that our policy increases the success rate by up to 61 percentage points over the baselines. Simulation and real robot experiments suggest that our policy learns to use the arm as a "tail" to initiate robot turning at high speeds and to stabilize the quadruped under external perturbations. Quantitatively, in simulation experiments, we cut the failure rate up to 43. 6% during high-speed turning and up to 31. 8% for quadruped under external forces compared to using a locked arm.

ICRA Conference 2024 Conference Paper

Open X-Embodiment: Robotic Learning Datasets and RT-X Models: Open X-Embodiment Collaboration

• Abby O'Neill
• Abdul Rehman
• Abhiram Maddukuri
• Abhishek Gupta 0004
• Abhishek Padalkar
• Abraham Lee
• Acorn Pooley
• Agrim Gupta

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x. github.io.

ICRA Conference 2024 Conference Paper

Orbit-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity

• Qinxi Yu
• Masoud Moghani
• Karthik Dharmarajan
• Vincent Schorp
• William Chung-Ho Panitch
• Jingzhou Liu
• Kush Hari
• Huang Huang

Physics-based simulations have accelerated progress in robot learning for driving, manipulation, and locomotion. Yet, a fast, accurate, and robust surgical simulation environment remains a challenge. In this paper, we present Orbit-Surgical, a physics-based surgical robot simulation framework with photorealistic rendering in NVIDIA Omniverse. We provide 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR) which represent common subtasks in surgical training. Orbit-Surgical leverages GPU parallelization to train reinforcement learning and imitation learning algorithms to facilitate study of robot learning to augment human surgical skills. Orbit-Surgical also facilitates realistic synthetic data generation for active perception tasks. We demonstrate Orbit-Surgical sim-to-real transfer of learned policies onto a physical dVRK robot. Project website: orbit-surgical. github.io

IROS Conference 2024 Conference Paper

SuFIA: Language-Guided Augmented Dexterity for Robotic Surgical Assistants

• Masoud Moghani
• Lars Doorenbos
• William Chung-Ho Panitch
• Sean Huver
• Mahdi Azizian
• Ken Goldberg
• Animesh Garg

In this work, we present SuFIA, the first framework for natural language-guided augmented dexterity for robotic surgical assistants. SuFIA incorporates the strong reasoning capabilities of large language models (LLMs) with perception modules to implement high-level planning and low-level control of a robot for surgical sub-task execution. This enables a learning-free approach to surgical augmented dexterity without any in-context examples or motion primitives. SuFIA uses a human-in-the-loop paradigm by restoring control to the surgeon in the case of insufficient information, mitigating unexpected errors for mission-critical tasks. We evaluate SuFIA on four surgical sub-tasks in a simulation environment and two sub-tasks on a physical surgical robotic platform in the lab, demonstrating its ability to perform common surgical sub-tasks through supervised autonomous operation under challenging physical and workspace conditions. Project website: orbit-surgical.github.io/sufia

ICRA Conference 2023 Conference Paper

AutoBag: Learning to Open Plastic Bags and Insert Objects

• Lawrence Yunliang Chen
• Baiyu Shi
• Daniel Seita
• Richard Cheng
• Thomas Kollar
• David Held
• Ken Goldberg

Thin plastic bags are ubiquitous in retail stores, healthcare, food handling, recycling, homes, and school lunchrooms. They are challenging both for perception (due to specularities and occlusions) and for manipulation (due to the dynamics of their 3D deformable structure). We formulate the task of “bagging: ” manipulating common plastic shopping bags with two handles from an unstructured initial state to an open state where at least one solid object can be inserted into the bag and lifted for transport. We propose a self-supervised learning framework where a dual-arm robot learns to recognize the handles and rim of plastic bags using UV-fluorescent markings; at execution time, the robot does not use UV markings or UV light. We propose the AutoBag algorithm, where the robot uses the learned perception model to open a plastic bag through iterative manipulation. We present novel metrics to evaluate the quality of a bag state and new motion primitives for reorienting and opening bags based on visual observations. In physical experiments, a YuMi robot using AutoBag is able to open bags and achieve a success rate of 16/30 for inserting at least one item across a variety of initial bag configurations. Supplementary material is available at https://sites.google.com/view/autobag.

ICRA Conference 2023 Conference Paper

Automating Vascular Shunt Insertion with the dVRK Surgical Robot

• Karthik Dharmarajan
• William Chung-Ho Panitch
• Muyan Jiang
• Kishore Srinivas
• Baiyu Shi
• Yahav Avigal
• Huang Huang
• Thomas Low

Vascular shunt insertion is a fundamental surgical procedure used to temporarily restore blood flow to tissues. It is often performed in the field after major trauma. We formulate a problem of automated vascular shunt insertion and propose a pipeline to perform Automated Vascular Shunt Insertion (AVSI) using a da Vinci Research Kit. The pipeline uses a learned visual model to estimate the locus of the vessel rim, plans a grasp on the rim, and moves to grasp at that point. The first robot gripper then pulls the rim to stretch open the vessel with a dilation motion. The second robot gripper then proceeds to insert a shunt into the vessel phantom (a model of the blood vessel) with a chamfer tilt followed by a screw motion. Results suggest that AVSI achieves a high success rate even with tight tolerances and varying vessel orientations up to 30°. Supplementary material, dataset, videos, and visualizations can be found at https://sites.google.com/berkeley.edu/autolab-avsi.

IROS Conference 2023 Conference Paper

Bagging by Learning to Singulate Layers Using Interactive Perception

• Lawrence Yunliang Chen
• Baiyu Shi
• Roy Lin
• Daniel Seita
• Ayah Ahmad
• Richard Cheng
• Thomas Kollar
• David Held

Many fabric handling and 2D deformable material tasks in homes and industries require singulating layers of material such as opening a bag or arranging garments for sewing. In contrast to methods requiring specialized sensing or end effectors, we use only visual observations with ordinary parallel jaw grippers. We propose SLIP: Singulating Layers using Interactive Perception, and apply SLIP to the task of autonomous bagging. We develop SLIP-Bagging, a bagging algorithm that manipulates a plastic or fabric bag from an unstructured state and uses SLIP to grasp the top layer of the bag to open it for object insertion. In physical experiments, a YuMi robot achieves a success rate of 67% to 81% across bags of a variety of materials, shapes, and sizes, significantly improving in success rate and generality over prior work. Experiments also suggest that SLIP can be applied to tasks such as singulating layers of folded cloth and garments. Supplementary material is available at https://sites.google.com/view/slip-bagging/.

ICRA Conference 2023 Conference Paper

Can Machines Garden? Systematically Comparing the AlphaGarden vs. Professional Horticulturalists

• Simeon Adebola
• Rishi Parikh
• Mark Presten
• Satvik Sharma
• Shrey Aeron
• Ananth Rao
• Sandeep Mukherjee
• Tomson Qu

The AlphaGarden is an automated testbed for indoor polyculture farming which combines a first-order plant simulator, a gantry robot, a seed planting algorithm, plant phenotyping and tracking algorithms, irrigation sensors and algorithms, and custom pruning tools and algorithms. In this paper, we systematically compare the performance of the AlphaGarden to professional horticulturalists on the staff of the UC Berkeley Oxford Tract Greenhouse. The humans and the machine tend side-by-side polyculture gardens with the same seed arrangement. We compare performance in terms of canopy coverage, plant diversity, and water consumption. Results from two 60-day cycles suggest that the automated AlphaGarden performs comparably to professional horticulturalists in terms of coverage and diversity, and reduces water consumption by as much as 44%. Code, videos, and datasets are available at https//sites.google.com/berkeley.edulsystematiccomparison

IROS Conference 2023 Conference Paper

FogROS2-SGC: A ROS2 Cloud Robotics Platform for Secure Global Connectivity

• Kaiyuan Chen 0001
• Ryan Hoque
• Karthik Dharmarajan
• Edith LLontop
• Simeon Adebola
• Jeffrey Ichnowski
• John Kubiatowicz
• Ken Goldberg

The Robot Operating System (ROS2) is the most widely used software platform for building robotics applications. FogROS2 extends ROS2 to allow robots to access cloud computing on demand. We introduce FogROS2-SGC, an extension of FogROS2 that can effectively connect robot systems across different physical locations, networks, and Data Distribution Services (DDS). With globally unique and location-independent identifiers, FogROS2-SGC can securely and efficiently route data between robotics components around the globe. FogROS2-SGC is agnostic to the ROS2 distribution and configuration, is compatible with non-ROS2 software, and seamlessly extends existing ROS2 applications without any code modification. We evaluate FogROS2-SGC with 4 robots and compute nodes that are 3600 km apart. Experiments suggest FogROS2-SGC is 19x faster than rosbridge (a ROS2 package with comparable features, but lacking security). Videos and code are available on the website https://sites.google.com/view/fogros2-sgc.

ICRA Conference 2023 Conference Paper

FogROS2: An Adaptive Platform for Cloud and Fog Robotics Using ROS 2

• Jeffrey Ichnowski
• Kaiyuan Chen 0001
• Karthik Dharmarajan
• Simeon Adebola
• Michael Danielczuk
• Víctor Mayoral Vilches
• Nikhil Jha
• Hugo Zhan

Mobility, power, and price points often dictate that robots do not have sufficient computing power on board to run contemporary robot algorithms at desired rates. Cloud computing providers such as AWS, GCP, and Azure offer immense computing power and increasingly low latency on demand, but tapping into that power from a robot is non-trivial. We present FogROS2, an open-source platform to facilitate cloud and fog robotics that is included in the Robot Operating System 2 (ROS 2) distribution. FogROS2 is distinct from its predecessor FogROS1 in 9 ways, including lower latency, overhead, and startup times; improved usability, and additional automation, such as region and computer type selection. Additionally, FogROS2 gains performance, timing, and additional improvements associated with ROS 2. In common robot applications, FogROS2 reduces SLAM latency by 50 %, reduces grasp planning time from 14 s to 1. 2 s, and speeds up motion planning 45x. When compared to FogROS1, FogROS2 reduces network utilization by up to 3. 8x, improves startup time by 63 %, and network round-trip latency by 97 % for images using video compression. The source code, examples, and documentation for FogROS2 are available at https://github.com/BerkeleyAutomation/FogROS2, and is available through the official ROS 2 repository at https://index.ros.org/p/FogROS2/.

IROS Conference 2023 Conference Paper

Learning to Efficiently Plan Robust Frictional Multi-Object Grasps

• Wisdom C. Agboh
• Satvik Sharma
• Kishore Srinivas
• Mallika Parulekar
• Gaurav Datta
• Tianshuang Qiu
• Jeffrey Ichnowski
• Eugen Solowjow

We consider a decluttering problem where multiple rigid convex polygonal objects rest in randomly placed positions and orientations on a planar surface and must be efficiently transported to a packing box using both single and multi-object grasps. Prior work considered frictionless multi-object grasping. In this paper, we introduce friction to increase the number of potential grasps for a given group of objects, and thus increase picks per hour. We train a neural network using real examples to plan robust multi-object grasps. In physical experiments, we find a 13. 7% increase in success rate, a 1. 6x increase in picks per hour, and a 6. 3x decrease in grasp planning time compared to prior work on multi-object grasping. Compared to single-object grasping, we find a 3. 1x increase in picks per hour.

ICRA Conference 2023 Conference Paper

Safe Self-Supervised Learning in Real of Visuo-Tactile Feedback Policies for Industrial Insertion

• Letian Fu
• Huang Huang
• Lars Berscheid
• Hui Li
• Ken Goldberg
• Sachin Chitta

Industrial insertion tasks are often performed repetitively with parts that are subject to tight tolerances and prone to breakage. Learning an industrial insertion policy in real is challenging as the collision between the parts and the environment can cause slippage or breakage of the part. In this paper, we present a safe self-supervised method to learn a visuo-tactile insertion policy that is robust to grasp pose variations. The method reduces human input and collisions between the part and the receptacle. The method divides the insertion task into two phases. In the first align phase, a tactile-based grasp pose estimation model is learned to align the insertion part with the receptacle. In the second insert phase, a vision-based policy is learned to guide the part into the receptacle. The robot uses force-torque sensing to achieve a safe self-supervised data collection pipeline. Physical experiments on the USB insertion task from the NIST Assembly Taskboard suggest that the resulting policies can achieve 45/45 insertion successes on 45 different initial grasp poses, improving on two baselines: (1) a behavior cloning agent trained on 50 human insertion demonstrations (1/45) and (2) an online RL policy (TD3) trained in real (0/45).

ICRA Conference 2023 Conference Paper

SGTM 2. 0: Autonomously Untangling Long Cables using Interactive Perception

• Kaushik Shivakumar
• Vainavi Viswanath
• Anrui Gu
• Yahav Avigal
• Justin Kerr
• Jeffrey Ichnowski
• Richard Cheng
• Thomas Kollar

Cables are commonplace in homes, hospitals, and industrial warehouses and are prone to tangling. This paper extends prior work on autonomously untangling long cables by introducing novel uncertainty quantification metrics and actions that interact with the cable to reduce perception uncertainty. We present Sliding and Grasping for Tangle Manipulation 2. 0 (SGTM 2. 0), a system that autonomously untangles cables approximately 3 meters in length with a bilateral robot using estimates of uncertainty at each step to inform actions. By interactively reducing uncertainty, SGTM 2. 0 significantly reduces run-time. Physical experiments with 84 trials suggest that SGTM $2. 0$ can achieve 83% untangling success on cables with 1 or 2 overhand and figure-8 knots, and 70% termination detection success across these configurations, outperforming SGTM 1. 0 by 43% in untangling accuracy and 200% in completion time. Supplementary material, visualizations, and videos can be found at sites.google.com/view/sgtm2.

NeurIPS Conference 2023 Conference Paper

Video Prediction Models as Rewards for Reinforcement Learning

• Alejandro Escontrela
• Ademi Adeniji
• Wilson Yan
• Ajay Jain
• Xue Bin Peng
• Ken Goldberg
• Youngwoon Lee
• Danijar Hafner

Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https: //ViperRL. com

IROS Conference 2022 Conference Paper

Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions

• Alejandro Escontrela
• Xue Bin Peng
• Wenhao Yu 0003
• Tingnan Zhang
• Atil Iscen
• Ken Goldberg
• Pieter Abbeel

Training a high-dimensional simulated agent with an under-specified reward function often leads the agent to learn physically infeasible strategies that are ineffective when deployed in the real world. To mitigate these unnatural behaviors, reinforcement learning practitioners often utilize complex reward functions that encourage physically plausible behaviors. However, a tedious labor-intensive tuning process is often required to create hand-designed rewards which might not easily generalize across platforms and tasks. We propose substituting complex reward functions with “style rewards” learned from a dataset of motion capture demonstrations. A learned style reward can be combined with an arbitrary task reward to train policies that perform tasks using naturalistic strategies. These natural strategies can also facilitate transfer to the real world. We build upon Adversarial Motion Priors - an approach from the computer graphics domain that encodes a style reward from a dataset of reference motions - to demonstrate that an adversarial approach to training policies can produce behaviors that transfer to a real quadrupedal robot without requiring complex reward functions. We also demonstrate that an effective style reward can be learned from a few seconds of motion capture data gathered from a German Shepherd and leads to energy-efficient locomotion strategies with natural gait transitions.

IROS Conference 2022 Conference Paper

All You Need is LUV: Unsupervised Collection of Labeled Images Using UV-Fluorescent Markings

• Brijen Thananjeyan
• Justin Kerr
• Huang Huang
• Joseph E. Gonzalez
• Ken Goldberg

Learning-based perception systems in robotics often requires large-scale image segmentation annotation. Current approaches rely on human labelers, which can be expensive, or simulation data, which can visually differ from real data. This paper proposes Labels from UltraViolet (LUV), a novel framework that enables rapid, automated, inexpensive, high quality data collection in real. LUV uses transparent, UV-fluorescent paint with programmable UV LEDs to collect paired images of a scene in standard and UV lighting. This makes it possible to autonomously extract segmentation masks and keypoints via color thresholding. We apply LUV to a suite of diverse robot perception tasks: locating fabric keypoints, cable segmentation, and surgical needle detection to evaluate its labeling quality, flexibility, and data collection rate. Results suggest that LUV is 180–2500 times faster than a human labeler across the tasks while retaining accuracy and strong task performance. Code, datasets, visualizations, and supplementary material can be found at https://sites.google.com/berkeley.edu/luv.

ICRA Conference 2022 Conference Paper

GOMP-FIT: Grasp-Optimized Motion Planning for Fast Inertial Transport

• Jeffrey Ichnowski
• Yahav Avigal
• Yi Liu
• Ken Goldberg

High-speed motions in pick-and-place operations are critical to making robots cost-effective in many automation scenarios, from warehouses and manufacturing to hospitals and homes. However, motions can be too fast-such as when the object being transported has an open-top, is fragile, or both. One way to avoid spills or damage, is to move the arm slowly. We propose an alternative: Grasp-Optimized Motion Planning for Fast Inertial Transport (GOMP-FIT), a time-optimizing motion planner based on our prior work, that includes con-straints based on accelerations at the robot end-effector. With GOMP-FIT, a robot can perform high-speed motions that avoid obstacles and use inertial forces to its advantage. In experiments transporting open-top containers with varying tilt tolerances, whereas GOMP computes sub-second motions that spill up to 90 % of the contents during transport, GOMP-FIT generates motions that spill 0 % of contents while being slowed by as little as 0 % when there are few obstacles, 30 % when there are high obstacles and 45-degree tolerances, and 50 % when there 15-degree tolerances and few obstacles. Videos and more at: https://berkeleyautomation.github.io/gomp-fit/.

ICRA Conference 2022 Conference Paper

Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learning

• Aditya Ganapathi
• Pete Florence
• Jake Varley
• Kaylee Burns
• Ken Goldberg
• Andy Zeng 0001

Action representation is an important yet often overlooked aspect in end-to-end robot learning with deep networks. Choosing one action space over another (e. g. target joint positions, or Cartesian end-effector poses) can result in surprisingly stark performance differences between various downstream tasks - and as a result, considerable research has been devoted to finding the right action space for a given application. However, in this work, we instead investigate how our models can discover and learn for themselves which action space to use. Leveraging recent work on implicit behavioral cloning, which takes both observations and actions as input, we demonstrate that it is possible to present the same action in multiple different spaces to the same policy - allowing it to learn inductive patterns from each space. Specifically, we study the benefits of combining Cartesian and joint action spaces in the context of learning manipulation skills. To this end, we present Implicit Kinematic Policies (IKP), which incorporates the kinematic chain as a differentiable module within the deep network. Quantitative experiments across several simulated continuous control tasks-from scooping piles of small objects, to lifting boxes with elbows, to precise block insertion with miscalibrated robots-suggest IKP not only learns complex prehensile and non-prehensile manipulation from pixels better than baseline alternatives, but also can learn to compensate for small joint encoder offset errors. Finally, we also run qualitative experiments on a real UR5e to demonstrate the feasibility of our algorithm on a physical robotic system with real data. See https://tinyurl.com/4wz3nf86 for code and supplementary material.

ICRA Conference 2022 Conference Paper

IPC-GraspSim: Reducing the Sim2Real Gap for Parallel-Jaw Grasping with the Incremental Potential Contact Model

• Chung Min Kim
• Michael Danielczuk
• Isabella Huang
• Ken Goldberg

Accurately simulating whether an object will be lifted securely or dropped during grasping is a longstanding Sim2Real challenge. Soft compliant jaw tips are almost universally used with parallel-jaw robot grippers due to their ability to increase contact area and friction between the jaws and the object to be manipulated. However, interactions between the compliant surfaces and rigid objects are notoriously difficult to model. We introduce IPC-GraspSim, a novel grasp simulator that extends Incremental Potential Contact (IPC) - a highly accurate collision + deformation model developed in 2020 for computer graphics. IPC-GraspSim models both the dynamics and the deformation of compliant jaw tips to reduce Sim2Real gap for robot grasping. We evaluate IPC-GraspSim using a set of 2, 000 physical grasps across 16 adversarial objects where analytic models perform poorly. In comparison to both analytic quasistatic contact models (soft point contact, REACH, 6DFC) and dynamic grasp simulators (Isaac Gym with FleX), results suggest IPC-GraspSim can predict robustness with higher precision and recall (F1 = 0. 85). IPC-GraspSim increases F1 score by 0. 03 to 0. 20 over analytic baselines and 0. 09 over Isaac Gym, at a cost of 8000x and 1. 5x more compute time, respectively. All data, code, videos, and supplementary material are available at https://sites.google.com/berkeley.edu/ipcgraspsim.

IROS Conference 2022 Conference Paper

Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research

• Ryan Hoque
• Kaushik Shivakumar
• Shrey Aeron
• Gabriel Deza
• Aditya Ganapathi
• Adrian Wong
• Johnny Lee
• Andy Zeng 0001

Autonomous fabric manipulation is a longstanding challenge in robotics, but evaluating progress is difficult due to the cost and diversity of robot hardware. Using Reach, a cloud robotics platform that enables low-latency remote execution of control policies on physical robots, we present the first systematic benchmarking of fabric manipulation al-gorithms on physical hardware. We develop 4 novel learning-based algorithms that model expert actions, keypoints, reward functions, and dynamic motions, and we compare these against 4 learning-free and inverse dynamics algorithms on the task of folding a crumpled T-shirt with a single robot arm. The entire lifecycle of data collection, model training, and policy evaluation was performed remotely without physical access to the robot workcell. Results suggest a new algorithm combining imitation learning with analytic methods achieves human-level performance on the flattening task and 93% of human-level performance on the folding task. See https://sites.google.com/berkeley.edu/cloudfolding for all data, code, models, and supplemental material.

ICRA Conference 2022 Conference Paper

Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles

• Albert Wilcox
• Justin Kerr
• Brijen Thananjeyan
• Jeffrey Ichnowski
• Minho Hwang
• Samuel Paradis
• Danyal M. Fer
• Ken Goldberg

Robotic Surgical Assistants (RSAs) are commonly used to perform minimally invasive surgeries by expert surgeons. However, long procedures filled with tedious and repetitive tasks such as suturing can lead to surgeon fatigue, motivating the automation of suturing. As visual tracking of a thin reflective needle is extremely challenging, prior work has modified the needle with nonreflective contrasting paint. As a step towards automation of a suturing subtask without modifying the needle, we propose HOUSTON: Handover of Unmodified, Surgical, Tool-Obstructed Needles, a problem and algorithm that uses a learned active sensing policy with a stereo camera to iteratively localize and align the needle into a visible and accessible pose for the other gripper. To compensate for robot positioning and needle perception errors, the algorithm then executes a high-precision grasping motion that uses multiple cameras. Physical experiments with the da Vinci Research Kit (dVRK) suggest a success rate of 96. 7% on needles used in training, and 75 - 92. 9% on needles unseen in training. On sequential handovers, HOUSTON successfully executes 32. 4 handovers on average before failure. To our knowledge, this work is the first to study handover of unmodified surgical needles. See https://tinyurl.com/houston-surgery for additional materials including details about offline datasets and model architectures.

ICRA Conference 2022 Conference Paper

LEGS: Learning Efficient Grasp Sets for Exploratory Grasping

• Letian Fu
• Michael Danielczuk
• Ashwin Balakrishna
• Daniel S. Brown
• Jeffrey Ichnowski
• Eugen Solowjow
• Ken Goldberg

While deep learning has enabled significant progress in designing general purpose robot grasping systems, there remain objects which still pose challenges for these systems. Recent work on Exploratory Grasping has formalized the problem of systematically exploring grasps on these adversarial objects and explored a multi-armed bandit model for identifying high-quality grasps on each object stable pose. However, these systems are still limited to exploring a small number or grasps on each object. We present Learned Efficient Grasp Sets (LEGS), an algorithm that efficiently explores thousands of possible grasps by maintaining small active sets of promising grasps and determining when it can stop exploring the object with high confidence. Experiments suggest that LEGS can identify a high-quality grasp more efficiently than prior algorithms which do not use active sets. In simulation experiments, we measure the gap between the success probability of the best grasp identified by LEGS, baselines, and the most-robust grasp (verified ground truth). After 3000 exploration steps, LEGS outperforms baseline algorithms on 10/14 and 25/39 objects on the Dex-Net Adversarial and EGAD! datasets respectively. We then evaluate LEGS in physical experiments; trials on 3 challenging objects suggest that LEGS converges to high-performing grasps significantly faster than baselines. See https://sites.google.com/view/LEGS-exp-grasping for supplemental material and videos.

ICRA Conference 2022 Conference Paper

Mechanical Search on Shelves using a Novel "Bluction" Tool

• Huang Huang
• Michael Danielczuk
• Chung Min Kim
• Letian Fu
• Zachary Tam
• Jeffrey Ichnowski
• Anelia Angelova
• Brian Ichter

Shelves are common in homes, warehouses, and commercial settings due to their storage efficiency. However, this efficiency comes at the cost of reduced visibility and accessibility. When looking from a side (lateral) view of a shelf, most objects will be fully occluded, resulting in a constrained lateral-access mechanical search problem. To address this problem, we introduce: (1) a novel bluction tool, which combines a thin pushing blade and a suction cup gripper, (2) a simulation pipeline and perception model that combine ray-casting with 2D Minkowski sums to efficiently generate target occupancy distributions, and (3) a novel search policy, which optimally reduces target object distribution support area using the bluction tool. Experimental data from 2000 simulated shelf trials and 18 trials with a physical Fetch robot suggest that a bluction tool can improve the average success rate by 26% in simulation and 67% in physical experiments over the highest-performing push-only policy.

NeurIPS Conference 2022 Conference Paper

Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations

• Albert Wilcox
• Ashwin Balakrishna
• Jules Dedieu
• Wyame Benslimane
• Daniel Brown
• Ken Goldberg

Providing densely shaped reward functions for RL algorithms is often exceedingly challenging, motivating the development of RL algorithms that can learn from easier-to-specify sparse reward functions. This sparsity poses new exploration challenges. One common way to address this problem is using demonstrations to provide initial signal about regions of the state space with high rewards. However, prior RL from demonstrations algorithms introduce significant complexity and many hyperparameters, making them hard to implement and tune. We introduce Monte Carlo Actor-Critic (MCAC), a parameter free modification to standard actor-critic algorithms which initializes the replay buffer with demonstrations and computes a modified $Q$-value by taking the maximum of the standard temporal distance (TD) target and a Monte Carlo estimate of the reward-to-go. This encourages exploration in the neighborhood of high-performing trajectories by encouraging high $Q$-values in corresponding regions of the state space. Experiments across $5$ continuous control domains suggest that MCAC can be used to significantly increase learning efficiency across $6$ commonly used RL and RL-from-demonstrations algorithms. See https: //sites. google. com/view/mcac-rl for code and supplementary material.

ICRA Conference 2022 Conference Paper

Real2Sim2Real: Self-Supervised Learning of Physical Single-Step Dynamic Actions for Planar Robot Casting

• Vincent Lim
• Huang Huang
• Lawrence Yunliang Chen
• Jonathan Wang
• Jeffrey Ichnowski
• Daniel Seita
• Michael Laskey
• Ken Goldberg

This paper introduces the task of Planar Robot Casting (PRC): where one planar motion of a robot arm holding one end of a cable causes the other end to slide across the plane toward a desired target. PRC allows the cable to reach points beyond the robot workspace and has applications for cable management in homes, warehouses, and factories. To efficiently learn a PRC policy for a given cable, we propose Real2Sim2Real, a self-supervised framework that automatically collects physical trajectory examples to tune parameters of a dynamics simulator using Differential Evolution, generates many simulated examples, and then learns a policy using a weighted combination of simulated and physical data. We evaluate Real2Sim2Real with three simulators, Isaac Gym-segmented, Isaac Gym-hybrid, and PyBullet, two function approximators, Gaussian Processes and Neural Networks (NNs), and three cables with differing stiffness, torsion, and friction. Results with 240 physical trials suggest that the PRC policies can attain median error distance (as % of cable length) ranging from 8 % to 14 %, outperforming baselines and policies trained on only real or only simulated examples. Code, data, and videos are available at https://tinyurl.com/robotcast.

IROS Conference 2022 Conference Paper

SpeedFolding: Learning Efficient Bimanual Folding of Garments

• Yahav Avigal
• Lars Berscheid
• Tamim Asfour
• Torsten Kröger
• Ken Goldberg

Folding garments reliably and efficiently is a long standing challenge in robotic manipulation due to the complex dynamics and high dimensional configuration space of garments. An intuitive approach is to initially manipulate the garment to a canonical smooth configuration before folding. In this work, we develop SpeedFolding, a reliable and efficient bimanual system, which given user-defined instructions as folding lines, manipulates an initially crumpled garment to (1) a smoothed and (2) a folded configuration. Our primary contribution is a novel neural network architecture that is able to predict pairs of gripper poses to parameterize a diverse set of bimanual action primitives. After learning from 4300 human- annotated and self-supervised actions, the robot is able to fold garments from a random initial configuration in under 120 s on average with a success rate of 93 %. Real-world experiments show that the system is able to generalize to unseen garments of different color, shape, and stiffness. While prior work achieved 3–6 Folds Per Hour (FPH), SpeedFolding achieves 30–40 FPH. See https://pantor.github.io/speedfolding for code, videos, and datasets.

IROS Conference 2021 Conference Paper

A Multi-Chamber Smart Suction Cup for Adaptive Gripping and Haptic Exploration

• Tae Myung Huh
• Kate Sanders 0002
• Michael Danielczuk
• Monica S. Li
• Lawrence Yunliang Chen
• Ken Goldberg
• Hannah S. Stuart

We present a novel robot end-effector for gripping and haptic exploration. Tactile sensing through suction flow monitoring is achieved with a new suction cup design that contains multiple chambers for air flow. Each chamber connects with its own remote pressure transducer, which enables both absolute and differential pressure measures between chambers. By changing the overall vacuum applied to this smart suction cup, it can perform different functions such as gentle haptic exploration (low pressure) and monitoring breaks in the seal during strong astrictive gripping (high pressure). Haptic exploration of surfaces through sliding and palpation can guide the selection of suction grasp locations and help to identify the local surface geometry. During suction gripping, a trained LSTM network can localize breaks in the suction seal between four quadrants with up to 97% accuracy and detects breaks in the suction seal early enough to avoid total grasp failure.

NeurIPS Conference 2021 Conference Paper

Accelerating Quadratic Optimization with Reinforcement Learning

• Jeffrey Ichnowski
• Paras Jain
• Bartolomeo Stellato
• Goran Banjac
• Michael Luo
• Francesco Borrelli
• Joseph E. Gonzalez
• Ion Stoica

First-order methods for quadratic optimization such as OSQP are widely used for large-scale machine learning and embedded optimal control, where many related problems must be rapidly solved. These methods face two persistent challenges: manual hyperparameter tuning and convergence time to high-accuracy solutions. To address these, we explore how Reinforcement Learning (RL) can learn a policy to tune parameters to accelerate convergence. In experiments with well-known QP benchmarks we find that our RL policy, RLQP, significantly outperforms state-of-the-art QP solvers by up to 3x. RLQP generalizes surprisingly well to previously unseen problems with varying dimension and structure from different applications, including the QPLIB, Netlib LP and Maros-M{\'e}sz{\'a}ros problems. Code, models, and videos are available at https: //berkeleyautomation. github. io/rlqp/.

IROS Conference 2021 Conference Paper

Disentangling Dense Multi-Cable Knots

• Vainavi Viswanath
• Jennifer Grannen
• Priya Sundaresan
• Brijen Thananjeyan
• Ashwin Balakrishna
• Ellen R. Novoseller
• Jeffrey Ichnowski
• Michael Laskey

Disentangling two or more cables often requires many steps to remove crossings between and within cables. We formalize the problem of disentangling multiple cables and present an algorithm, Iterative Reduction Of Non-planar Multiple cAble kNots (IRON-MAN), that outputs robot actions to remove crossings from multi-cable knotted structures. IRON-MAN uses a learned perception system inspired by prior work in single-cable untying to imitate a graph-based supervisor, and operates on RGB image inputs of the workspace. Given a sequence of images as input, the system can disentangle two-cable twists, three-cable braids, and knots of two or three cables, such as overhand, square, carrick bend, sheet bend, crown, and fisherman’s knots. IRON-MAN keeps track of task-relevant keypoints corresponding to cable endpoints and crossings and iteratively disentangles the cables by identifying and undoing crossings that are critical to knot structure. Using a da Vinci surgical robot, we experimentally evaluate the effectiveness of IRON-MAN on disentangling multi-cable knots of types that appear in the training data, as well as generalizing to novel classes of multi-cable knots. Results suggest that IRON-MAN is effective in disentangling knots involving up to three cables with 80. 5% success and generalizing to knot types that are not present during training, with cables of identical or distinct colors. Supplementary material and videos can be found at https://tinyurl.com/multi-cable-disentangling.

ICRA Conference 2021 Conference Paper

Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation

• Samuel Paradis
• Minho Hwang
• Brijen Thananjeyan
• Jeffrey Ichnowski
• Daniel Seita
• Danyal M. Fer
• Thomas Low
• Joseph E. Gonzalez

Assisting surgeons with automation of surgical subtasks is challenging due to backlash, hysteresis, and variable tensioning in cable-driven robots. These issues are exacerbated as surgical instruments are changed during an operation. In this work, we propose a framework for automation of high- precision surgical subtasks by learning local, sample-efficient, accurate, closed-loop policies that use visual feedback instead of robot encoder estimates. This framework, which we call deep Intermittent Visual Servoing (IVS), switches to a learned visual servo policy for high-precision segments of repetitive surgical tasks while relying on a coarse open-loop policy for the segments where precision is not necessary. We train the policy using only 180 human demonstrations that are roughly 2 seconds each. Results on a da Vinci Research Kit suggest that combining the coarse policy with half a second of corrections from the learned policy during each high-precision segment improves the success rate on the Fundamentals of Laparoscopic Surgery peg transfer task from 72. 9% to 99. 2%, 31. 3% to 99. 2%, and 47. 2% to 100. 0% for 3 instruments with differing cable properties. In the contexts we studied, IVS attains the highest published success rates for automated surgical peg transfer and is significantly more reliable than previous techniques when instruments are changed. Supplementary material is available at https://tinyurl.com/ivs-icra.

ICRA Conference 2021 Conference Paper

Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics

• Aditya Ganapathi
• Priya Sundaresan
• Brijen Thananjeyan
• Ashwin Balakrishna
• Daniel Seita
• Jennifer Grannen
• Minho Hwang
• Ryan Hoque

Robotic fabric manipulation is challenging due to the infinite dimensional configuration space, self-occlusion, and complex dynamics of fabrics. There has been significant prior work on learning policies for specific fabric manipulation tasks, but comparatively less focus on algorithms which can perform many different tasks. We take a step towards this goal by learning point-pair correspondences across different fabric configurations in simulation. Then, given a single demonstration of a new task from an initial fabric configuration, these correspondences can be used to compute geometrically equivalent actions in a new fabric configuration. This makes it possible to define policies to robustly imitate a broad set of multi-step fabric smoothing and folding tasks. The resulting policies achieve 80. 3% average task success rate across 10 fabric manipulation tasks on two different physical robotic systems. Results also suggest robustness to fabrics of various colors, sizes, and shapes. See https://tinyurl.com/fabric-descriptors for supplementary material and videos.

ICRA Conference 2021 Conference Paper

Learning Seed Placements and Automation Policies for Polyculture Farming with Companion Plants

• Yahav Avigal
• Anna Deza
• William Wong
• Sebastian Oehme
• Mark Presten
• Mark Theis
• Jackson Chui
• Paul Shao

Polyculture farming is a sustainable farming technique based on synergistic interactions between differing plant types that make them more resistant to diseases and pests and better able to retain water. Reduced uniformity can reduce use of pesticides, fertilizer, and water, but is more labor intensive and more challenging to automate. We describe a scaled physical testbed (1. 5m×3. 0m) that uses a high resolution camera and soil sensors to monitor polyculture plants to facilitate tuning of plant growth, companion effects, and irrigation parameters for a first-order garden simulator. We use this simulator to develop a novel seed placement algorithm that increases coverage and diversity, and a learned pruning policy. In simulation experiments, the seed placement algorithm yields 60% more coverage and 10% more diversity than random seed placement and the learned pruning policy runs 1000X faster than a procedural lookahead policy to achieve high leaf coverage and plant diversity on adversarial gardens that include plant species with diverse growth rates. These models and policies provide the groundwork for a fully-automated system under development. Code, datasets and supplementary material can be found at https://github.com/BerkeleyAutomation/AlphaGarden/.

ICRA Conference 2021 Conference Paper

Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks

• Daniel Seita
• Pete Florence
• Jonathan Tompson
• Erwin Coumans
• Vikas Sindhwani
• Ken Goldberg
• Andy Zeng 0001

Rearranging and manipulating deformable objects such as cables, fabrics, and bags is a long-standing challenge in robotic manipulation. The complex dynamics and high-dimensional configuration spaces of deformables, compared to rigid objects, make manipulation difficult not only for multi-step planning, but even for goal specification. Goals cannot be as easily specified as rigid object poses, and may involve complex relative spatial relations such as "place the item inside the bag". In this work, we develop a suite of simulated benchmarks with 1D, 2D, and 3D deformable structures, including tasks that involve image-based goal-conditioning and multi-step deformable manipulation. We propose embedding goal-conditioning into Transporter Networks, a recently proposed model architecture for learning robotic manipulation that rearranges deep features to infer displacements that can represent pick and place actions. We demonstrate that goal-conditioned Transporter Networks enable agents to manipulate deformable structures into flexibly specified configurations without test-time visual anchors for target locations. We also significantly extend prior results using Transporter Networks for manipulating deformable objects by testing on tasks with 2D and 3D deformables. Supplementary material is available at https://berkeleyautomation.github.io/bags/.

IROS Conference 2021 Conference Paper

Mechanical Search on Shelves using Lateral Access X-RAY

• Huang Huang
• Marcus Dominguez-Kuhne
• Vishal Satish
• Michael Danielczuk
• Kate Sanders 0002
• Jeffrey Ichnowski
• Andrew Lee 0002
• Anelia Angelova

Finding an occluded object in a lateral access environment such as a shelf or cabinet is a problem that arises in many contexts such as warehouses, retail, healthcare, shipping, and homes. While this problem, known as mechanical search, is well-studied in overhead access environments, lateral access environments introduce constraints on the poses of objects and on available grasp actions, and pushing actions are preferred to preserve the environment structure. We propose LAX-RAY (Lateral Access maXimal Reduction in support Area of occupancY distribution): a system that combines target object occupancy distribution prediction with a mechanical search policy that sequentially pushes occluding objects to reveal a given target object. For scenarios with extruded polygonal objects, we introduce two lateral-access search policies that encode a history of predicted target distributions and can plan up to three actions into the future. We introduce a First-Order Shelf Simulator (FOSS) and use it to evaluate these policies in 800 simulated random shelf environments per policy. We also evaluate in 5 physical shelf environments using a Fetch robot with an embedded PrimeSense RGBD Camera and an attached pushing blade. The policies outperform baselines by up to 25% in simulation and up to 60% in physical experiments. Additionally, the two-step prediction policy is the highest performing in simulation for 8 objects with a 69% success rate, suggesting a tradeoff between future information and prediction errors. Code, videos, and supplementary material can be found at https://sites.google.com/berkeley.edu/lax-ray.

ICML Conference 2021 Conference Paper

Policy Gradient Bayesian Robust Optimization for Imitation Learning

• Zaynah Javed
• Daniel S. Brown
• Satvik Sharma
• Jerry Zhu
• Ashwin Balakrishna
• Marek Petrik
• Anca D. Dragan
• Ken Goldberg

The difficulty in specifying rewards for many real-world problems has led to an increased focus on learning rewards from human feedback, such as demonstrations. However, there are often many different reward functions that explain the human feedback, leaving agents with uncertainty over what the true reward function is. While most policy optimization approaches handle this uncertainty by optimizing for expected performance, many applications demand risk-averse behavior. We derive a novel policy gradient-style robust optimization approach, PG-BROIL, that optimizes a soft-robust objective that balances expected performance and risk. To the best of our knowledge, PG-BROIL is the first policy optimization algorithm robust to a distribution of reward hypotheses which can scale to continuous MDPs. Results suggest that PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse and outperforms state-of-the-art imitation learning algorithms when learning from ambiguous demonstrations by hedging against uncertainty, rather than seeking to uniquely identify the demonstrator’s reward function.

ICML Conference 2021 Conference Paper

Resource Allocation in Multi-armed Bandit Exploration: Overcoming Sublinear Scaling with Adaptive Parallelism

• Brijen Thananjeyan
• Kirthevasan Kandasamy
• Ion Stoica
• Michael I. Jordan
• Ken Goldberg
• Joseph E. Gonzalez

We study exploration in stochastic multi-armed bandits when we have access to a divisible resource that can be allocated in varying amounts to arm pulls. We focus in particular on the allocation of distributed computing resources, where we may obtain results faster by allocating more resources per pull, but might have reduced throughput due to nonlinear scaling. For example, in simulation-based scientific studies, an expensive simulation can be sped up by running it on multiple cores. This speed-up however, is partly offset by the communication among cores, which results in lower throughput than if fewer cores were allocated to run more trials in parallel. In this paper, we explore these trade-offs in two settings. First, in a fixed confidence setting, we need to find the best arm with a given target success probability as quickly as possible. We propose an algorithm which trades off between information accumulation and throughput and show that the time taken can be upper bounded by the solution of a dynamic program whose inputs are the gaps between the sub-optimal and optimal arms. We also prove a matching hardness result. Second, we present an algorithm for a fixed deadline setting, where we are given a time deadline and need to maximize the probability of finding the best arm. We corroborate our theoretical insights with simulation experiments that show that the algorithms consistently match or outperform baseline algorithms on a variety of problem instances.

ICRA Conference 2021 Conference Paper

Robots of the Lost Arc: Self-Supervised Learning to Dynamically Manipulate Fixed-Endpoint Cables

• Harry Zhang
• Jeffrey Ichnowski
• Daniel Seita
• Jonathan Wang
• Huang Huang
• Ken Goldberg

We explore how high-speed robot arm motions can dynamically manipulate ropes and cables to vault over obstacles, knock objects from pedestals, and weave between obstacles. In this paper, we propose a self-supervised learning framework that enables a UR5 robot to perform these three tasks. The framework finds a 3D apex point for the robot arm, which, together with a task-specific trajectory function, defines an arcing motion that dynamically manipulates the cable to perform a task with varying obstacle and target locations. The trajectory function computes minimum-jerk motions that are constrained to remain within joint limits and to travel through the 3D apex point by repeatedly solving quadratic programs to find the shortest and fastest feasible motion. We experiment with 5 physical cables with different thickness and mass and compare performance against two baselines in which a human chooses the apex point. Results suggest that a baseline with a fixed apex across the three tasks achieves respective success rates of 51. 7 %, 36. 7 %, and 15. 0 %, and a baseline with human-specified, task-specific apex points achieves 66. 7 %, 56. 7 %, and 15. 0 % success rate respectively, while the robot using the learned apex point can achieve success rates of 81. 7 % in vaulting, 65. 0 % in knocking, and 60. 0 % in weaving. Code, data, and supplementary materials are available at https://sites.google.com/berkeley.edu/dynrope/home.

ICRA Conference 2021 Conference Paper

Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search

• Andrey Kurenkov
• Roberto Martín-Martín
• Jeffrey Ichnowski
• Ken Goldberg
• Silvio Savarese

Searching for objects in indoor organized environments such as homes or offices is part of our everyday activities. When looking for a desired object, we reason about the rooms and containers the object is likely to be in; the same type of container will have a different probability of containing the target depending on which room it is in. We also combine geometric and semantic information to infer what container is best to search, or what other objects are best to move, if the target object is hidden from view. We use a 3D scene graph representation to capture the hierarchical, semantic, and geometric aspects of this problem. To exploit this representation in a search process, we introduce Hierarchical Mechanical Search (HMS), a method that guides an agent’s actions towards finding a target object specified with a natural language description. HMS is based on a novel neural network architecture that uses neural message passing of vectors with visual, geometric, and linguistic information to allow HMS to process data across layers of the graph while combining semantic and geometric cues. HMS is trained on 1000 3D scene graphs and evaluated on a novel dataset of 500 3D scene graphs with dense placements of semantically related objects in storage locations, and is shown to be significantly better than several baselines at finding objects. It is also close to the oracle policy in terms of the median number of actions required. Additional qualitative results can be found at https://ai.stanford.edu/mech-search/hms

ICRA Conference 2021 Conference Paper

Serverless Multi-Query Motion Planning for Fog Robotics

• Raghav Anand
• Jeffrey Ichnowski
• Chenggang Wu 0001
• Joseph M. Hellerstein
• Joseph E. Gonzalez
• Ken Goldberg

Robots in semi-structured environments such as homes and warehouses sporadically require computation of high-dimensional motion plans. Cloud and fog-based parallelization of motion planning can speed up planning. This can be further made efficient by the use of "serverless" on-demand computing as opposed to always-on high end computers. This paper explores parallelizing the computation of a sampling-based multi-query motion planner based on asymptotically-optimal Probabilistic Road Maps (PRM*) using the simultaneous execution of 100s of cloud-based serverless functions. We propose an algorithm to overcome the communication and bandwidth limitations of serverless computing and use different work-sharing techniques to further optimize the cost and run time. Additionally, we provide proofs of probabilistic completeness and asymptotic optimality. In experiments on synthetic benchmarks and on a physical Fetch robot performing a sequence of decluttering motions, we observe up to a 50x speedup relative to a 4 core edge computer with only a marginally higher cost.

ICRA Conference 2020 Conference Paper

6DFC: Efficiently Planning Soft Non-Planar Area Contact Grasps using 6D Friction Cones

• Jingyi Xu
• Michael Danielczuk
• Eckehard G. Steinbach
• Ken Goldberg

Analytic grasp planning algorithms typically approximate compliant contacts with soft point contact models to compute grasp quality, but these models are overly conservative and do not capture the full range of grasps available. While area contact models can reduce the number of false negatives predicted by point contact models, they have been restricted to a 3D analysis of the wrench applied at the contact and so are still overly conservative. We extend traditional 3D friction cones and present an efficient algorithm for calculating the 6D friction cone (6DFC) for a non-planar area contact between a compliant gripper and a rigid object. We introduce a novel sampling algorithm to find the 6D friction limit surface for a non-planar area contact and a linearization method for these ellipsoids that reduces the computation of 6DFC constraints to a quadratic program. We show that constraining the wrench applied at the contact in this way increases recall, a metric inversely related to the number of false negative predictions, by 17% and precision, a metric inversely related to the number of false positive predictions, by 2% over soft point contact models on results from 1500 physical grasps on 12 3D printed nonplanar objects with an ABB YuMi robot. The 6DFC algorithm also achieves 6% higher recall with similar precision and 85x faster runtime than a previously proposed area contact model.

IROS Conference 2020 Conference Paper

Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor

• Daniel Seita
• Aditya Ganapathi
• Ryan Hoque
• Minho Hwang
• Edward Cen
• Ajay Kumar Tanwani
• Ashwin Balakrishna
• Brijen Thananjeyan

Sequential pulling policies to flatten and smooth fabrics have applications from surgery to manufacturing to home tasks such as bed making and folding clothes. Due to the complexity of fabric states and dynamics, we apply deep imitation learning to learn policies that, given color (RGB), depth (D), or combined color-depth (RGBD) images of a rectangular fabric sample, estimate pick points and pull vectors to spread the fabric to maximize coverage. To generate data, we develop a fabric simulator and an algorithmic supervisor that has access to complete state information. We train policies in simulation using domain randomization and dataset aggregation (DAgger) on three tiers of difficulty in the initial randomized configuration. We present results comparing five baseline policies to learned policies and report systematic comparisons of RGB vs D vs RGBD images as inputs. In simulation, learned policies achieve comparable or superior performance to analytic baselines. In 180 physical experiments with the da Vinci Research Kit (dVRK) surgical robot, RGBD policies trained in simulation attain coverage of 83% to 95% depending on difficulty tier, suggesting that effective fabric smoothing policies can be learned from an algorithmic supervisor and that depth sensing is a valuable addition to color alone. Supplementary material is available at https://sites.google.com/view/fabric-smoothing.

ICRA Conference 2020 Conference Paper

Dex-Net AR: Distributed Deep Grasp Planning Using a Commodity Cellphone and Augmented Reality App

• Harry Zhang
• Jeffrey Ichnowski
• Yahav Avigal
• Joseph E. Gonzalez
• Ion Stoica
• Ken Goldberg

Consumer demand for augmented reality (AR) in mobile phone applications, such as the Apple ARKit. Such applications have potential to expand access to robot grasp planning systems such as Dex-Net. AR apps use structure from motion methods to compute a point cloud from a sequence of RGB images taken by the camera as it is moved around an object. However, the resulting point clouds are often noisy due to estimation errors. We present a distributed pipeline, Dex-Net AR, that allows point clouds to be uploaded to a server in our lab, cleaned, and evaluated by Dex-Net grasp planner to generate a grasp axis that is returned and displayed as an overlay on the object. We implement Dex-Net AR using the iPhone and ARKit and compare results with those generated with high-performance depth sensors. The success rates with AR on harder adversarial objects are higher than traditional depth images. The server URL is https://sites.google.com/berkeley.edu/dex-net-ar/home.

ICRA Conference 2020 Conference Paper

Fog Robotics Algorithms for Distributed Motion Planning Using Lambda Serverless Computing

• Jeffrey Ichnowski
• William Lee
• Victor Murta
• Samuel Paradis
• Ron Alterovitz
• Joseph E. Gonzalez
• Ion Stoica
• Ken Goldberg

For robots using motion planning algorithms such as RRT and RRT*, the computational load can vary by orders of magnitude as the complexity of the local environment changes. To adaptively provide such computation, we propose Fog Robotics algorithms in which cloud-based serverless lambda computing provides parallel computation on demand. To use this parallelism, we propose novel motion planning algorithms that scale effectively with an increasing number of serverless computers. However, given that the allocation of computing is typically bounded by both monetary and time constraints, we show how prior learning can be used to efficiently allocate resources at runtime. We demonstrate the algorithms and application of learned parallel allocation in both simulation and with the Fetch commercial mobile manipulator using Amazon Lambda to complete a sequence of sporadically computationally intensive motion planning tasks.

ICRA Conference 2020 Conference Paper

GOMP: Grasp-Optimized Motion Planning for Bin Picking

• Jeffrey Ichnowski
• Michael Danielczuk
• Jingyi Xu
• Vishal Satish
• Ken Goldberg

Rapid and reliable robot bin picking is a critical challenge in automating warehouses, often measured in picks-per-hour (PPH). We explore increasing PPH using faster motions based on optimizing over a set of candidate grasps. The source of this set of grasps is two-fold: (1) grasp-analysis tools such as Dex-Net generate multiple candidate grasps, and (2) each of these grasps has a degree of freedom about which a robot gripper can rotate. In this paper, we present Grasp-Optimized Motion Planning (GOMP), an algorithm that speeds up the execution of a bin-picking robot's operations by incorporating robot dynamics and a set of candidate grasps produced by a grasp planner into an optimizing motion planner. We compute motions by optimizing with sequential quadratic programming (SQP) and iteratively updating trust regions to account for the non-convex nature of the problem. In our formulation, we constrain the motion to remain within the mechanical limits of the robot while avoiding obstacles. We further convert the problem to a time-minimization by repeatedly shorting a time horizon of a trajectory until the SQP is infeasible. In experiments with a UR5, GOMP achieves a speedup of 9x over a baseline planner.

ICRA Conference 2020 Conference Paper

Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data

• Priya Sundaresan
• Jennifer Grannen
• Brijen Thananjeyan
• Ashwin Balakrishna
• Michael Laskey
• Kevin Stone
• Joseph E. Gonzalez
• Ken Goldberg

Robotic manipulation of deformable 1D objects such as ropes, cables, and hoses is challenging due to the lack of high-fidelity analytic models and large configuration spaces. Furthermore, learning end-to-end manipulation policies directly from images and physical interaction requires significant time on a robot and can fail to generalize across tasks. We address these challenges using interpretable deep visual representations for rope, extending recent work on dense object descriptors for robot manipulation. This facilitates the design of interpretable and transferable geometric policies built on top of the learned representations, decoupling visual reasoning and control. We present an approach that learns point-pair correspondences between initial and goal rope configurations, which implicitly encodes geometric structure, entirely in simulation from synthetic depth images. We demonstrate that the learned representation — dense depth object descriptors (DDODs) — can be used to manipulate a real rope into a variety of different arrangements either by learning from demonstrations or using interpretable geometric policies. In 50 trials of a knot-tying task with the ABB YuMi Robot, the system achieves a 66% knot-tying success rate from previously unseen configurations. See https://tinyurl.com/rope-learning for supplementary material and videos.

ICRA Conference 2020 Conference Paper

Minimal Work: A Grasp Quality Metric for Deformable Hollow Objects

• Jingyi Xu
• Michael Danielczuk
• Jeffrey Ichnowski
• Jeffrey Mahler
• Eckehard G. Steinbach
• Ken Goldberg

Robot grasping of deformable hollow objects such as plastic bottles and cups is challenging, as the grasp should resist disturbances while minimally deforming the object so as not to damage it or dislodge liquids. We propose minimal work as a novel grasp quality metric that combines wrench resistance and object deformation. We introduce an efficient algorithm to compute the work required to resist an external wrench for a manipulation task by solving a linear program. The algorithm first computes the minimum required grasp force and an estimation of the gripper jaw displacements based on the object's empirical stiffness at different locations. The work done by the jaws is the product of the grasp force and the displacements. Grasps requiring minimal work are considered to be of high quality. We collect 460 physical grasps with a UR5 robot and a Robotiq gripper. We consider a grasp to be successful if it completes the task without damaging the object or dislodging the content. Physical experiments suggest that the minimal work quality metric reaches 74. 2% balanced accuracy, a metric that is the raw accuracy normalized by the number of successful and failed real-world grasps, and is up to 24. 2% higher than classical wrench-based quality metrics.

ICRA Conference 2020 Conference Paper

Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos

• Ajay Kumar Tanwani
• Pierre Sermanet
• Andy Yan
• Raghav Anand
• Mariano Phielipp
• Ken Goldberg

Learning meaningful visual representations in an embedding space can facilitate generalization in downstream tasks such as action segmentation and imitation. In this paper, we learn a motion-centric representation of surgical video demonstrations by grouping them into action segments/subgoals/options in a semi-supervised manner. We present Motion2Vec, an algorithm that learns a deep embedding feature space from video observations by minimizing a metric learning loss in a Siamese network: images from the same action segment are pulled together while pushed away from randomly sampled images of other segments, while respecting the temporal ordering of the images. The embeddings are iteratively segmented with a recurrent neural network for a given parametrization of the embedding space after pre-training the Siamese network. We only use a small set of labeled video segments to semantically align the embedding space and assign pseudo-labels to the remaining unlabeled data by inference on the learned model parameters. We demonstrate the use of this representation to imitate surgical suturing kinematic motions from publicly available videos of the JIGSAWS dataset. Results give 85. 5% segmentation accuracy on average suggesting performance improvement over several state-of-the-art baselines, while kinematic pose imitation gives 0. 94 centimeter error in position per observation on the test set. Videos, code and data are available at: https://sites.google.com/view/motion2vec.

IROS Conference 2020 Conference Paper

X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions

• Michael Danielczuk
• Anelia Angelova
• Vincent Vanhoucke
• Ken Goldberg

For applications in e-commerce, warehouses, healthcare, and home service, robots are often required to search through heaps of objects to grasp a specific target object. For mechanical search, we introduce X-Ray, an algorithm based on learned occupancy distributions. We train a neural network using a synthetic dataset of RGBD heap images labeled for a set of standard bounding box targets with varying aspect ratios. X-Ray minimizes support of the learned distribution as part of a mechanical search policy in both simulated and real environments. We benchmark these policies against two baseline policies on 1, 000 heaps of 15 objects in simulation where the target object is partially or fully occluded. Results suggest that X-Ray is significantly more efficient, as it succeeds in extracting the target object 82% of the time, 15% more often than the best-performing baseline. Experiments on an ABB YuMi robot with 20 heaps of 25 household objects suggest that the learned policy transfers easily to a physical system, where it outperforms baseline policies by 15% in success rate with 17% fewer actions. Datasets, videos, and experiments are available at https://sites.google.com/berkeley.edu/x-ray.

ICRA Conference 2019 Conference Paper

A Fog Robotic System for Dynamic Visual Servoing

• Nan Tian
• Ajay Kumar Tanwani
• Jinfa Chen
• Mas Ma
• Robert Zhang 0001
• Bill Huang
• Ken Goldberg
• Somayeh Sojoudi

Cloud Robotics is a paradigm where multiple robots are connected to cloud services via Internet to access “unlimited” computation power, at the cost of network communication. However, due to limitations such as network latency and variability, it is difficult to control dynamic, human compliant service robots directly from the cloud. In this work, we combine cloud robotics with an agile edge device to build a Fog Robotic system by leveraging an asynchronous protocol with a “heartbeat” signal. We use the system to enable robust teleoperation of a dynamic self-balancing robot from the cloud. We use the system to pick up boxes from static locations, a task commonly performed in warehouse logistics. To make cloud teleoperation more intuitive and efficient, we program a cloud-based image based visual servoing (IBVS) module to automatically assist the cloud teleoperator during the object pickups. Visual feedbacks, including apriltag recognition and tracking, are performed in the cloud to emulate a Fog Robotic object recognition system for IBVS. We demonstrate the feasibility of a dynamic real-time automation system using this cloud-edge hybrid design, which opens up possibilities of deploying dynamic robotic control with deep-learning recognition systems in Fog Robotics. Finally, we show that Fog Robotics enables the self-balancing service robot to pick up a box automatically from a person under unstructured environments.

ICRA Conference 2019 Conference Paper

A Fog Robotics Approach to Deep Robot Learning: Application to Object Recognition and Grasp Planning in Surface Decluttering

• Ajay Kumar Tanwani
• Nitesh Mor
• John Kubiatowicz
• Joseph E. Gonzalez
• Ken Goldberg

The growing demand of industrial, automotive and service robots presents a challenge to the centralized Cloud Robotics model in terms of privacy, security, latency, bandwidth, and reliability. In this paper, we present a `Fog Robotics' approach to deep robot learning that distributes compute, storage and networking resources between the Cloud and the Edge in a federated manner. Deep models are trained on non-private (public) synthetic images in the Cloud; the models are adapted to the private real images of the environment at the Edge within a trusted network and subsequently, deployed as a service for low-latency and secure inference/prediction for other robots in the network. We apply this approach to surface decluttering, where a mobile robot picks and sorts objects from a cluttered floor by learning a deep object recognition and a grasp planning model. Experiments suggest that Fog Robotics can improve performance by sim-to-real domain adaptation in comparison to exclusively using Cloud or Edge resources, while reducing the inference cycle time by 4× to successfully declutter 86% of objects over 213 attempts.

ICRA Conference 2019 Conference Paper

Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

• Michael Danielczuk
• Andrey Kurenkov
• Ashwin Balakrishna
• Matthew Matl
• David Wang
• Roberto Martín-Martín
• Animesh Garg
• Silvio Savarese

When operating in unstructured environments such as warehouses, homes, and retail centers, robots are frequently required to interactively search for and retrieve specific objects from cluttered bins, shelves, or tables. Mechanical Search describes the class of tasks where the goal is to locate and extract a known target object. In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin. The robot uses an RGBD perception system and control policies to iteratively select, parameterize, and perform one of 3 actions - push, suction, grasp - until the target object is extracted, or either a time limit is exceeded, or no high confidence push or grasp is available. We present a study of 5 algorithmic policies for mechanical search, with 15, 000 simulated trials and 300 physical trials for heaps ranging from 10 to 20 objects. Results suggest that success can be achieved in this long-horizon task with algorithmic policies in over 95% of instances and that the number of actions required scales approximately linearly with the size of the heap. Code and supplementary material can be found at http://ai.stanford.edu/mech-search.

IROS Conference 2019 Conference Paper

Partial Caging: A Clearance-Based Definition and Deep Learning

• Anastasiia Varava
• Michael C. Welle
• Jeffrey Mahler
• Ken Goldberg
• Danica Kragic
• Florian T. Pokomy

Caging grasps limit the mobility of an object to a bounded component of configuration space. We introduce a notion of partial cage quality based on maximal clearance of an escaping path. As this is a computationally demanding task even in a two-dimensional scenario, we propose a deep learning approach. We design two convolutional neural networks and construct a pipeline for real-time partial cage quality estimation directly from 2D images of object models and planar caging tools. One neural network, CageMaskNN, is used to identify caging tool locations that can support partial cages, while a second network that we call CageClearanceNN is trained to predict the quality of those configurations. A dataset of 3811 images of objects and more than 19 million caging tool configurations is used to train and evaluate these networks on previously unseen objects and caging tool configurations. Furthermore, the networks are trained jointly on configurations for both 3 and 4 caging tool configurations whose shape varies along a 1-parameter family of increasing elongation. In experiments, we study how the networks’ performance depends on the size of the training dataset, as well as how to efficiently deal with unevenly distributed training data. In further analysis, we show that the evaluation pipeline can approximately identify connected regions of successful caging tool placements and we evaluate the continuity of the cage quality score evaluation along caging tool trajectories. Experiments show that evaluation of a given configuration on a GeForce GTX 1080 GPU takes less than 6 ms.

ICRA Conference 2019 Conference Paper

Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data

• Michael Danielczuk
• Matthew Matl
• Saurabh Gupta 0001
• Andrew Li
• Andrew Lee 0002
• Jeffrey Mahler
• Ken Goldberg

The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive hand-labeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can transfer successfully to the real world. We present a method for automated dataset generation and rapidly generate a synthetic training dataset of 50, 000 depth images and 320, 000 object masks using simulated heaps of 3D CAD models. We train a variant of Mask R-CNN with domain randomization on the generated dataset to perform category-agnostic instance segmentation without any hand-labeled data and we evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution depth images of challenging, densely-cluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall on COCO benchmarks, and achieves performance levels similar to a Mask R-CNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE.

ICRA Conference 2018 Conference Paper

Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

• Tianhao Zhang 0001
• Zoe McCarthy
• Owen Jow
• Dennis Lee 0003
• Xi Chen 0022
• Ken Goldberg
• Pieter Abbeel

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.

ICRA Conference 2018 Conference Paper

Dex-Net 3. 0: Computing Robust Vacuum Suction Grasp Targets in Point Clouds Using a New Analytic Model and Deep Learning

• Jeffrey Mahler
• Matthew Matl
• Jason Xinyu Liu
• Albert Li
• David V. Gealy
• Ken Goldberg

Vacuum-based end effectors are widely used in industry and are often preferred over parallel-jaw and multifinger grippers due to their ability to lift objects with a single point of contact. Suction grasp planners often target planar surfaces on point clouds near the estimated centroid of an object. In this paper, we propose a compliant suction contact model that computes the quality of the seal between the suction cup and local target surface and a measure of the ability of the suction grasp to resist an external gravity wrench. To characterize grasps, we estimate robustness to perturbations in end-effector and object pose, material properties, and external wrenches. We analyze grasps across 1, 500 3D object models to generate Dex-Net 3. 0, a dataset of 2. 8 million point clouds, suction grasps, and grasp robustness labels. We use Dex-Net 3. 0 to train a Grasp Quality Convolutional Neural Network (GQ-CNN) to classify robust suction targets in point clouds containing a single object. We evaluate the resulting system in 350 physical trials on an ABB YuMi fitted with a pneumatic suction gripper. When evaluated on novel objects that we categorize as Basic (prismatic or cylindrical), Typical (more complex geometry), and Adversarial (with few available suction-grasp points) Dex-Net 3. 0 achieves success rates of 98%, 82%, and 58% respectively, improving to 81% in the latter case when the training set includes only adversarial objects. Code, datasets, and supplemental material can be found at http://berkeleyautomation.github.io/dex-net.

ICRA Conference 2018 Conference Paper

Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure

• Daniel Seita
• Sanjay Krishnan
• Roy Fox
• Stephen McKinley
• John F. Canny
• Ken Goldberg

Automating precision subtasks such as debridement (removing dead or diseased tissue fragments) with Robotic Surgical Assistants (RSAs) such as the da Vinci Research Kit (dVRK) is challenging due to inherent nOnlinearities in cable-driven systems. We propose and evaluate a novel two-phase coarse-to-fine calibration method. In Phase I (coarse), we place a red calibration marker on the end effector and let it randomly move through a set of open-loop trajectories to obtain a large sample set of camera pixels and internal robot end-effector configurations. This coarse data is then used to train a Deep Neural Network (DNN) to learn the coarse transformation bias. In Phase II (fine), the bias from Phase I is applied to move the end -effector toward a small set of specific target points on a printed sheet. For each target, a human operator manually adjusts the end -effector position by direct contact (not through teleoperation) and the residual compensation bias is recorded. This fine data is then used to train a Random Forest (RF) to learn the fine transformation bias. Subsequent experiments suggest that without calibration, position errors average 4. 55mm. Phase I can reduce average error to 2. 14mm and the combination of Phase I and Phase II can reduces average error to 1. 08mm. We apply these results to debridement of raisins and pumpkin seeds as fragment phantoms. Using an endoscopic stereo camera with standard edge detection, experiments with 120 trials achieved average success rates of 94. 5 %, exceeding prior results with much larger fragments (89. 4%) and achieving a speedup of 2. 1x, decreasing time per fragment from 15. 8 seconds to 7. 3 seconds. Source code, data, and videos are available at https://sites.google.com/view/calib-icra/.

ICML Conference 2018 Conference Paper

RLlib: Abstractions for Distributed Reinforcement Learning

• Eric Liang
• Richard Liaw
• Robert Nishihara
• Philipp Moritz
• Roy Fox
• Ken Goldberg
• Joseph E. Gonzalez
• Michael I. Jordan

Reinforcement learning (RL) algorithms involve the deep nesting of highly irregular computation patterns, each of which typically exhibits opportunities for distributed computation. We argue for distributing RL components in a composable way by adapting algorithms for top-down hierarchical control, thereby encapsulating parallelism and resource requirements within short-running compute tasks. We demonstrate the benefits of this principle through RLlib: a library that provides scalable software primitives for RL. These primitives enable a broad range of algorithms to be implemented with high performance, scalability, and substantial code reuse. RLlib is available as part of the open source Ray project at http: //rllib. io/.

ICRA Conference 2018 Conference Paper

Robustly Adjusting Indoor Drip Irrigation Emitters with the Toyota HSR Robot

• Ron Berenstein
• Roy Fox
• Stephen McKinley
• Stefano Carpin
• Ken Goldberg

Indoor plants in homes and commercial buildings such as malls, offices, airports, and hotels, can benefit from precision irrigation to maintain healthy growth and reduce water consumption. As active valves are too costly, and ongoing precise manual adjustment of drip emitters is impractical, we explore how the Toyota HSR mobile manipulator robot can autonomously adjust low-cost passive emitters. To provide sufficient accuracy for gripper alignment, we designed a lightweight, modular Emitter Localization Device (ELD) with cameras and LEDs that can be non-invasively mounted on the arm. This paper presents details of the design, algorithms, and experiments with adjusting emitters using a two-phase procedure: (1) aligning the robot base using the build-in hand camera, and (2) aligning the gripper axis with the emitter axis using the ELD. We report success rates and sensitivity analysis to tune computer vision parameters and joint motor gains. Experiments suggest that emitters can be adjusted with 95 % success rate in approximately 20 seconds.

ICRA Conference 2018 Conference Paper

Routing Algorithms for Robot Assisted Precision Irrigation

• Thomas C. Thayer
• Stavros Vougioukas
• Ken Goldberg
• Stefano Carpin

When robots navigate through vineyards to perform irrigation adjustments, an optimization problem emerges whereby robots are tasked with performing adjustments having the highest cumulative outcome within a given temporal budget due to limited battery charge. To this end, the robot needs to reach a set of spatially distributed sites, and the specific structure of the vineyard imposes various constraints on possible motions. In this paper we first demonstrate that this type of orienteering problem remains NP-hard even for the restricted class of graphs associated with precision irrigation. Then, we devise and analyze two greedy heuristics informed by the problem we consider. Finally, these algorithms are evaluated on settings associated with a commercial vineyard and we show that our methods favorably compare to solutions proposed in the past.

IROS Conference 2018 Conference Paper

Towards a Soft Fingertip with Integrated Sensing and Actuation

• Benjamin W. McInroe
• Carolyn L. Chen
• Ken Goldberg
• Ruzena Bajcsy
• Ronald S. Fearing

Soft material robots are attractive for safe interaction with humans and unstructured environments due to their compliance and low intrinsic stiffness and mass. These properties enable new capabilities such as the ability to conform to environmental geometry for tactile sensing and to undergo large shape changes for actuation. Due to the complex coupling between sensing and actuation in high-dimensional nonlinear soft systems, prior work in soft robotics has primarily focused on either sensing or actuation. This paper presents SOFTcell, a novel controllable stiffness tactile device that incorporates both optical sensing and pneumatic actuation. We report details on the device's design and implementation and analyze results from characterization experiments on sensitivity and performance, which show that SOFTcell can controllably increase its effective modulus from 4. 4kPa to 46. 1kPa. Additionally, we demonstrate the utility of SOFTcell for grasping in a reactive control task in which tactile data is used to detect fingertip shear as a grasped object slips, and cell pressurization is used to prevent the slip without the need to adjust fingertip position.

ICRA Conference 2017 Conference Paper

A cloud robot system using the dexterity network and berkeley robotics and automation as a service (Brass)

• Nan Tian
• Matthew Matl
• Jeffrey Mahler
• Yu Xiang Zhou
• Samantha Staszak
• Christopher Correa
• Steven Zheng
• Qiang Li

In support of Cloud Robotics, Robotics and Automation as a Service (RAaaS) frameworks have the potential to reduce the complexity of software development, simplify software installation and maintenance, and facilitate data sharing for machine learning. In this proof-of-concept paper, we describe Berkeley Robotics and Automation as a Service (Brass), a RAaaS prototype that allows robots to access a remote server that hosts a robust grasp-planning system (Dex-Net 1. 0) that maintains data on hundreds of candidate grasps on thousands of 3D object meshes and uses perturbation sampling to estimate and update a stochastic robustness metric for each grasp. Results suggest that such a system can increase grasp reliability over naive locally-computed grasping strategies with network latencies of 30 and 200 msec for servers 500 and 6000 miles away, respectively. We also study how the system can use execution reports from robots in the field to update grasp recommendations over time.

ICRA Conference 2017 Conference Paper

Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations

• Michael Laskey
• Caleb Chuck
• Jonathan Lee 0002
• Jeffrey Mahler
• Sanjay Krishnan
• Kevin Jamieson 0001
• Anca D. Dragan
• Ken Goldberg

Motivated by recent advances in Deep Learning for robot control, this paper considers two learning algorithms in terms of how they acquire demonstrations from fallible human supervisors. Human-Centric (HC) sampling is a standard supervised learning algorithm, where a human supervisor demonstrates the task by teleoperating the robot to provide trajectories consisting of state-control pairs. Robot-Centric (RC) sampling is an increasingly popular alternative used in algorithms such as DAgger, where a human supervisor observes the robot execute a learned policy and provides corrective control labels for each state visited. We suggest RC sampling can be challenging for human supervisors and prone to mislabeling. RC sampling can also induce error in policy performance because it repeatedly visits areas of the state space that are harder to learn. Although policies learned with RC sampling can be superior to HC sampling for standard learning models such as linear SVMs, policies learned with HC sampling may be comparable to RC when applied to expressive learning models such as deep learning and hyper-parametric decision trees, which can achieve very low training error provided there is enough data. We compare HC and RC using a grid world environment and a physical robot singulation task. In the latter the input is a binary image of objects on a planar worksurface and the policy generates a motion in the gripper to separate one object from the rest. We observe in simulation that for linear SVMs, policies learned with RC outperformed those learned with HC but that using deep models this advantage disappears. We also find that with RC, the corrective control labels provided by humans can be highly inconsistent. We prove there exists a class of examples in which at the limit, HC is guaranteed to converge to an optimal policy while RC may fail to converge. These results suggest a form of HC sampling may be preferable for highly-expressive learning models and human supervisors.

ICRA Conference 2017 Conference Paper

Design of parallel-jaw gripper tip surfaces for robust grasping

• Menglong Guo
• David V. Gealy
• Jacky Liang
• Jeffrey Mahler
• Aimee Goncalves
• Stephen McKinley
• Juan Aparicio Ojea
• Ken Goldberg

Parallel-jaw robot grippers can grasp almost any object and are ubiquitous in industry. Although the shape, texture, and compliance of gripper jaw surfaces affect grasp robustness, almost all commercially available grippers provide a pair of rectangular, planar, rigid jaw surfaces. Practitioners often modify these surfaces with a variety of ad-hoc methods such as adding rubber caps and/or wrapping with textured tape. This paper explores data-driven optimization of gripper jaw surfaces over a design space based on shape, texture, and compliance using rapid prototyping. In total, 37 jaw surface design variations were created using 3D printed casting molds and silicon rubber. The designs were evaluated with 1377 physical grasp experiments using a 4-axis robot (with automated reset). These tests evaluate grasp robustness as the probability that the jaws will acquire, lift, and hold a training set of objects at nominal grasp configurations computed by Dex-Net 1. 0. Hill-climbing in parameter space yielded a grid pattern of 0. 03 inch void depth and 0. 0375 inch void width on a silicone polymer with durometer of A30. We then evaluated performance of this design using an ABB YuMi robot grasping a set of eight difficult-to-grasp 3D printed objects in 80 grasps with four gripper surfaces. The factory-provided gripper tips succeeded in 28. 7% of the 80 trials, increasing to 68. 7% when the tips were wrapped with tape. Gripper tips with gecko-inspired surfaces succeeded in 80. 0% of trials, and gripper tips with the designed silicone surfaces succeeded in 93. 7% of trials.

ICRA Conference 2017 Conference Paper

Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning

• Brijen Thananjeyan
• Animesh Garg
• Sanjay Krishnan
• Carolyn Chen
• Lauren Miller
• Ken Goldberg

In the Fundamentals of Laparoscopic Surgery (FLS) standard medical training regimen, the Pattern Cutting task requires residents to demonstrate proficiency by maneuvering two tools, surgical scissors and tissue gripper, to accurately cut a circular pattern on surgical gauze suspended at the corners. Accuracy of cutting depends on tensioning, wherein the gripper pinches a point on the gauze in R 3 and pulls to induce and maintain tension in the material as cutting proceeds. An automated tensioning policy maps the current state of the gauze to output a direction of pulling as an action. The optimal tensioning policy depends on both the choice of pinch point and cutting trajectory. We explore the problem of learning a tensioning policy conditioned on specific cutting trajectories. Every timestep, we allow the gripper to react to the deformation of the gauze and progress of the cutting trajectory with a translation unit vector along an allowable set of directions. As deformation is difficult to analytically model and explicitly observe, we leverage deep reinforcement learning with direct policy search methods to learn tensioning policies using a finite-element simulator and then transfer them to a physical system. We compare the Deep RL tensioning policies with fixed and analytic (opposing the error vector with a fixed pinch point) policies on a set of 17 open and closed curved contours in simulation and 4 patterns in physical experiments with the da Vinci Research Kit (dVRK). Our simulation results suggest that learning to tension with Deep RL can significantly improve performance and robustness to noise and external forces.

ICRA Conference 2016 Conference Paper

Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization

• Siddarth Sen
• Animesh Garg
• David V. Gealy
• Stephen McKinley
• Yiming Jen
• Ken Goldberg

For supervised automation of multi-throw suturing in Robot-Assisted Minimally Invasive Surgery, we present a novel mechanical needle guide and a framework for optimizing needle size, trajectory, and control parameters using sequential convex programming. The Suture Needle Angular Positioner (SNAP) results in a 3x error reduction in the needle pose estimate in comparison with the standard actuator. We evaluate the algorithm and SNAP on a da Vinci Research Kit using tissue phantoms and compare completion time with that of humans from the JIGSAWS dataset [5]. Initial results suggest that the dVRK can perform suturing at 30% of human speed while completing 86% suture throws attempted. Videos and data are available at: berkeleyautomation.github.io/amts

ICRA Conference 2016 Conference Paper

Dex-Net 1. 0: A cloud-based network of 3D objects for robust grasp planning using a Multi-Armed Bandit model with correlated rewards

• Jeffrey Mahler
• Florian T. Pokorny
• Brian Hou
• Melrose Roderick
• Michael Laskey
• Mathieu Aubry
• Kai Kohlhoff
• Torsten Kröger

This paper presents the Dexterity Network (Dex-Net) 1. 0, a dataset of 3D object models and a sampling-based planning algorithm to explore how Cloud Robotics can be used for robust grasp planning. The algorithm uses a Multi- Armed Bandit model with correlated rewards to leverage prior grasps and 3D object models in a growing dataset that currently includes over 10, 000 unique 3D object models and 2. 5 million parallel-jaw grasps. Each grasp includes an estimate of the probability of force closure under uncertainty in object and gripper pose and friction. Dex-Net 1. 0 uses Multi-View Convolutional Neural Networks (MV-CNNs), a new deep learning method for 3D object classification, to provide a similarity metric between objects, and the Google Cloud Platform to simultaneously run up to 1, 500 virtual cores, reducing experiment runtime by up to three orders of magnitude. Experiments suggest that correlated bandit techniques can use a cloud-based network of object models to significantly reduce the number of samples required for robust grasp planning. We report on system sensitivity to variations in similarity metrics and in uncertainty in pose and friction. Code and updated information is available at http://berkeleyautomation.github.io/dex-net/.

ICRA Conference 2016 Conference Paper

High-dimensional Winding-Augmented Motion Planning with 2D topological task projections and persistent homology

• Florian T. Pokorny
• Danica Kragic
• Lydia E. Kavraki
• Ken Goldberg

Recent progress in motion planning has made it possible to determine homotopy inequivalent trajectories between an initial and terminal configuration in a robot configuration space. Current approaches have however either assumed the knowledge of differential one-forms related to a skeletonization of the collision space, or have relied on a simplicial representation of the free space. Both of these approaches are currently however not yet practical for higher dimensional configuration spaces. We propose 2D topological task projections (TTPs): mappings from the configuration space to 2-dimensional spaces where simplicial complex filtrations and persistent homology can identify topological properties of the high-dimensional free configuration space. Our approach only requires the availability of collision free samples to identify winding centers that can be used to determine homotopy inequivalent trajectories. We propose the Winding Augmented RRT and RRT* (WA-RRT/RRT*) algorithms using which homotopy inequivalent trajectories can be found. We evaluate our approach in experiments with configuration spaces of planar linkages with 2–10 degrees of freedom. Results indicate that our approach can reliably identify suitable topological task projections and our proposed WA-RRT and WA-RRT* algorithms were able to identify a collection of homotopy inequivalent trajectories in each considered configuration space dimension.

IROS Conference 2016 Conference Paper

Occlusion-aware multi-robot 3D tracking

• Karol Hausman
• Gregory Kahn
• Sachin Patil
• Jörg Müller 0004
• Ken Goldberg
• Pieter Abbeel
• Gaurav S. Sukhatme

We introduce an optimization-based control approach that enables a team of robots to cooperatively track a target using onboard sensing. In this setting, the robots are required to estimate their own positions as well as concurrently track the target. Our probabilistic method generates controls that minimize the expected uncertainty of the target. Additionally, our method efficiently reasons about occlusions between robots and takes them into account for the control generation. We evaluate our approach in a number of experiments in which we simulate a team of quadrotor robots flying in three-dimensional space to track a moving target on the ground. We compare our method to other state-of-the-art approaches represented by the random sampling technique, lattice planning method, and our previous method. Our experimental results indicate that our method achieves up to 8 times smaller maximum tracking error and up to 2 times smaller average tracking error than the next best approach in the presented scenarios.

ICRA Conference 2016 Conference Paper

SHIV: Reducing supervisor burden in DAgger using support vectors for efficient learning from demonstrations in high dimensional state spaces

• Michael Laskey
• Samantha Staszak
• Wesley Yu-Shu Hsieh
• Jeffrey Mahler
• Florian T. Pokorny
• Anca D. Dragan
• Ken Goldberg

Online learning from demonstration algorithms such as DAgger can learn policies for problems where the system dynamics and the cost function are unknown. However they impose a burden on supervisors to respond to queries each time the robot encounters new states while executing its current best policy. The MMD-IL algorithm reduces supervisor burden by filtering queries with insufficient discrepancy in distribution and maintaining multiple policies. We introduce the SHIV algorithm (Svm-based reduction in Human InterVention), which converges to a single policy and reduces supervisor burden in non-stationary high dimensional state distributions. To facilitate scaling and outlier rejection, filtering is based on a measure of risk defined in terms of distance to an approximate level set boundary defined by a One Class support vector machine. We report on experiments in three contexts: 1) a driving simulator with a 27, 936 dimensional visual feature space, 2) a push-grasping in clutter simulation with a 22 dimensional state space, and 3) physical surgical needle insertion with a 16 dimensional state space. Results suggest that SHIV can efficiently learn policies with up to 70% fewer queries that DAgger.

ICRA Conference 2016 Conference Paper

Topological trajectory clustering with relative persistent homology

• Florian T. Pokorny
• Ken Goldberg
• Danica Kragic

Cloud Robotics techniques based on Learning from Demonstrations suggest promising alternatives to manual programming of robots and autonomous vehicles. One challenge is that demonstrated trajectories may vary dramatically: it can be very difficult, if not impossible, for a system to learn control policies unless the trajectories are clustered into meaningful consistent subsets. Metric clustering methods, based on a distance measure, require quadratic time to compute a pairwise distance matrix and do not naturally distinguish topologically distinct trajectories. This paper presents an algorithm for topological clustering based on relative persistent homology, which, for a fixed underlying simplicial representation and discretization of trajectories, requires only linear time in the number of trajectories. The algorithm incorporates global constraints formalized in terms of the topology of sublevel or superlevel sets of a function and can be extended to incorporate probabilistic motion models. In experiments with real automobile and ship GPS trajectories as well as pedestrian trajectories extracted from video, the algorithm clusters trajectories into meaningful consistent subsets and, as we show in an experiment with ship trajectories, results in a faster and more efficient clustering than a metric clustering by Fréchet distance.

ICRA Conference 2016 Conference Paper

TSC-DL: Unsupervised trajectory segmentation of multi-modal surgical demonstrations with Deep Learning

• Adithyavairavan Murali
• Animesh Garg
• Sanjay Krishnan
• Florian T. Pokorny
• Pieter Abbeel
• Trevor Darrell
• Ken Goldberg

The growth of robot-assisted minimally invasive surgery has led to sizable datasets of fixed-camera video and kinematic recordings of surgical subtasks. Segmentation of these trajectories into locally-similar contiguous sections can facilitate learning from demonstrations, skill assessment, and salvaging good segments from otherwise inconsistent demonstrations. Manual, or supervised, segmentation can be prone to error and impractical for large datasets. We present Transition State Clustering with Deep Learning (TSC-DL), a new unsupervised algorithm that leverages video and kinematic data for task-level segmentation, and finds regions of the visual feature space that correlate with transition events using features constructed from layers of pre-trained image classification Deep Convolutional Neural Networks (CNNs). We report results on three datasets comparing Deep Learning architectures (AlexNet and VGG), choice of convolutional layer, dimensionality reduction techniques, visual encoding, and the use of Scale Invariant Feature Transforms (SIFT). We find that the deep architectures extract features that result in up-to a 30. 4% improvement in Silhouette Score (a measure of cluster tightness) over the traditional “shallow” features from SIFT. We also present cases where TSC-DL discovers human annotator omissions. Supplementary material, data and code is available at: http://berkeleyautomation.github.io/tsc-dl/

IROS Conference 2015 Conference Paper

A paced shared-control teleoperated architecture for supervised automation of multilateral surgical tasks

• Kamran Shamaei
• Yuhang Che
• Adithyavairavan Murali
• Siddarth Sen
• Sachin Patil
• Ken Goldberg
• Allison M. Okamura

Automation of repetitive tasks can improve laparoscopic surgical procedures by unloading surgeons and reducing duration, trauma, and expense. However, surgical procedures involve delicate manipulation of deformable tissues in a very dynamic environment, suggesting that automated execution of surgical tasks should be carried out under the supervision of the surgeon. We propose a teleoperated architecture that allows a surgeon to employ and supervise agents that can autonomously perform or assist with surgical tasks. The architecture is independent of the automation method. It includes a dominance factor that allows the surgeon to take control over the slave robot at any time, and an aggressiveness factor that sets the performance pace of the autonomous agent. We tested the architecture during execution of a multilateral tension-and-cut task, where a human operator and an autonomous agent are responsible for tensioning or cutting of a tissue. The architecture allowed for supervised and paced automation of the task. We found that collaboration of the human operator and autonomous agent can lead to shorter completion time compared to performance of only a human.

ICRA Conference 2015 Conference Paper

Active exploration using trajectory optimization for robotic grasping in the presence of occlusions

• Gregory Kahn
• Peter Sujan
• Sachin Patil
• Shaunak D. Bopardikar
• Julian Ryde
• Ken Goldberg
• Pieter Abbeel

We consider the task of actively exploring unstructured environments to facilitate robotic grasping of occluded objects. Typically, the geometry and locations of these objects are not known a priori. We mount an RGB-D sensor on the robot gripper to maintain a 3D voxel map of the environment during exploration. The objective is to plan the motion of the sensor in order to search for feasible grasp handles that lie within occluded regions of the map. In contrast to prior work that generates exploration trajectories by sampling, we directly optimize the exploration trajectory to find grasp handles. Since it is challenging to optimize over the discrete voxel map, we encode the uncertainty of the positions of the occluded grasp handles as a mixture of Gaussians, one per occluded region. Our trajectory optimization approach encourages exploration by penalizing a measure of the uncertainty. We then plan a collision-free trajectory for the robot arm to the detected grasp handle. We evaluated our approach by actively exploring and attempting 300 grasps. Our experiments suggest that compared to the baseline method of sampling 10 trajectories, which successfully grasped 58% of the objects, our active exploration formulation with trajectory optimization successfully grasped 93% of the objects, was 1. 3× faster, and had 3. 2× fewer failed grasp attempts.

ICRA Conference 2015 Conference Paper

GP-GPIS-OPT: Grasp planning with shape uncertainty using Gaussian process implicit surfaces and Sequential Convex Programming

• Jeffrey Mahler
• Sachin Patil
• Ben Kehoe
• Jur van den Berg
• Matei Ciocarlie
• Pieter Abbeel
• Ken Goldberg

Computing grasps for an object is challenging when the object geometry is not known precisely. In this paper, we explore the use of Gaussian process implicit surfaces (GPISs) to represent shape uncertainty from RGBD point cloud observations of objects. We study the use of GPIS representations to select grasps on previously unknown objects, measuring grasp quality by the probability of force closure. Our main contribution is GP-GPIS-OPT, an algorithm for computing grasps for parallel-jaw grippers on 2D GPIS object representations. Specifically, our method optimizes an approximation to the probability of force closure subject to antipodal constraints on the parallel jaws using Sequential Convex Programming (SCP). We also introduce GPIS-Blur, a method for visualizing 2D GPIS models based on blending shape samples from a GPIS. We test the algorithm on a set of 8 planar objects with transparency, translucency, and specularity. Our experiments suggest that GP-GPIS-OPT computes grasps with higher probability of force closure than a planner that does not consider shape uncertainty on our test objects and may converge to a grasp plan up to 5. 7×faster than using Monte-Carlo integration, a common method for grasp planning under shape uncertainty. Furthermore, initial experiments on the Willow Garage PR2 robot suggest that grasps selected with GP-GPIS-OPT are up to 90% more successful than those planned assuming a deterministic shape. Our dataset, code, and videos of our experiments are available at http://rll.berkeley.edu/icra2015grasping/.

ICRA Conference 2015 Conference Paper

Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms

• Adithyavairavan Murali
• Siddarth Sen
• Ben Kehoe
• Animesh Garg
• Seth McFarland
• Sachin Patil
• W. Douglas Boyd
• Susan Lim

Automating repetitive surgical subtasks such as suturing, cutting and debridement can reduce surgeon fatigue and procedure times and facilitate supervised tele-surgery. Programming is difficult because human tissue is deformable and highly specular. Using the da Vinci Research Kit (DVRK) robotic surgical assistant, we explore a “Learning By Observation” (LBO) approach where we identify, segment, and parameterize motion sequences and sensor conditions to build a finite state machine (FSM) for each subtask. The robot then executes the FSM repeatedly to tune parameters and if necessary update the FSM structure. We evaluate the approach on two surgical subtasks: debridement of 3D Viscoelastic Tissue Phantoms (3d-DVTP), in which small target fragments are removed from a 3D viscoelastic tissue phantom; and Pattern Cutting of 2D Orthotropic Tissue Phantoms (2d-PCOTP), a step in the standard Fundamentals of Laparoscopic Surgery training suite, in which a specified circular area must be cut from a sheet of orthotropic tissue phantom. We describe the approach and physical experiments with repeatability of 96% for 50 trials of the 3d-DVTP subtask and 70% for 20 trials of the 2d-PCOTP subtask. A video is available at: http://j.mp/Robot-Surgery-Video-Oct-2014.

IROS Conference 2015 Conference Paper

Models of human-centered automation in a debridement task

• Kirk A. Nichols
• Adithyavairavan Murali
• Siddarth Sen
• Ken Goldberg
• Allison M. Okamura

In robot-assisted surgery, manipulation tasks can be achieved through collaboration among robotic and human agents. Collaboration models can potentially include multiple agents working towards a shared objective - a scenario referred to as multilateral manipulation. In this work, we examine multilateral manipulation in the task of debridement: removing dead tissue or damaged tissue fragments to allow remaining healthy tissue to heal. We extended our previously developed multilateral manipulation software framework to the task of debridement and implemented four different collaboration models: (1) fully autonomous debridement, (2) shared control between a human and robotic agent, (3) supervised control where the operator identifies foreign bodies to be excised, and (4) teleoperation. We demonstrate these collaboration models with the RAVEN-II robot, an open-architecture surgical robot with two cable-driven 7-DOF arms. Each collaboration model included methods to remove foreign bodies from the field of view of a stereoscopic camera setup. Results indicate tradeoffs in experiment duration, hardware requirements, and safety implications among the four collaboration models.

ICRA Conference 2014 Conference Paper

Autonomous multilateral debridement with the Raven surgical robot

• Ben Kehoe
• Gregory Kahn
• Jeffrey Mahler
• Jonathan Kim
• Alex X. Lee
• Anna Lee
• Keisuke Nakagawa
• Sachin Patil

Autonomous robot execution of surgical sub-tasks has the potential to reduce surgeon fatigue and facilitate supervised tele-surgery. This paper considers the sub-task of surgical debridement: removing dead or damaged tissue fragments to allow the remaining healthy tissue to heal. We present an autonomous multilateral surgical debridement system using the Raven, an open-architecture surgical robot with two cable-driven 7 DOF arms. Our system combines stereo vision for 3D perception with trajopt, an optimization-based motion planner, and model predictive control (MPC). Laboratory experiments involving sensing, grasping, and removal of 120 fragments suggest that an autonomous surgical robot can achieve robustness comparable to human performance. Our robot system demonstrated the advantage of multilateral systems, as the autonomous execution was 1. 5× faster with two arms than with one; however, it was two to three times slower than a human. Execution speed could be improved with better state estimation that would allow more travel between MPC steps and fewer MPC replanning cycles. The three primary contributions of this paper are: (1) introducing debridement as a sub-task of interest for surgical robotics, (2) demonstrating the first reliable autonomous robot performance of a surgical sub-task using the Raven, and (3) reporting experiments that highlight the importance of accurate state estimation for future research. Further information including code, photos, and video is available at: http://rll.berkeley.edu/raven.

ICRA Conference 2014 Conference Paper

Gaussian belief space planning with discontinuities in sensing domains

• Sachin Patil
• Yan Duan
• John Schulman
• Ken Goldberg
• Pieter Abbeel

Discontinuities in sensing domains are common when planning for many robotic navigation and manipulation tasks. For cameras and 3D sensors, discontinuities may be inherent in sensor field of view or may change over time due to occlusions that are created by moving obstructions and movements of the sensor. The associated gaps in sensor information due to missing measurements pose a challenge for belief space and related optimization-based planning methods since there is no gradient information when the system state is outside the sensing domain. We address this in a belief space context by considering the signed distance to the sensing region. We smooth out sensing discontinuities by assuming that measurements can be obtained outside the sensing region with noise levels depending on a sigmoid function of the signed distance. We sequentially improve the continuous approximation by increasing the sigmoid slope over an outer loop to find plans that cope with sensor discontinuities. We also incorporate the information contained in not obtaining a measurement about the state during execution by appropriately truncating the Gaussian belief state. We present results in simulation for tasks with uncertainty involving navigation of mobile robots and reaching tasks with planar robot arms. Experiments suggest that the approach can be used to cope with discontinuities in sensing domains by effectively re-planning during execution.

ICRA Conference 2014 Conference Paper

Planning locally optimal, curvature-constrained trajectories in 3D using sequential convex optimization

• Yan Duan
• Sachin Patil
• John Schulman
• Ken Goldberg
• Pieter Abbeel

3D curvature-constrained motion planning finds applications in a wide variety of domains, including motion planning for flexible, bevel-tip medical needles, planning curvature-constrained channels in 3D printed implants for targeted brachytherapy dose delivery or channels for cooling turbine blades, and path planning for unmanned aerial vehicles (UAVs). In this work, we present a motion planning technique using sequential convex optimization for computing locally optimal, curvature-constrained trajectories to desired targets while avoiding obstacles in 3D environments. We report two main contributions in this work: (i) curvature-constrained trajectory optimization in 6D pose (position and orientation) space, and (ii) planning multiple trajectories that are mutually collision-free. We demonstrate the performance of our approach on two clinically motivated applications. Our experiments indicate that our approach can compute high-quality plans for medical needle steering in 1. 6 seconds on a commodity PC, enabling re-planning during execution to correct for perturbations. Our approach can also be used for designing optimized channel layouts within 3D printed implants for intracavitary brachytherapy.

AIJ Journal 2014 Journal Article

Potential-based bounded-cost search and Anytime Non-Parametric A ⁎

• Roni Stern
• Ariel Felner
• Jur van den Berg
• Rami Puzis
• Rajat Shah
• Ken Goldberg

This paper presents two new search algorithms: Potential Search (PTS) and Anytime Potential Search/Anytime Non-Parametric A ⁎ ( APTS/ANA ⁎ ). Both algorithms are based on a new evaluation function that is easy to implement and does not require user-tuned parameters. PTS is designed to solve bounded-cost search problems, which are problems where the task is to find as fast as possible a solution under a given cost bound. APTS/ANA ⁎ is a non-parametric anytime search algorithm discovered independently by two research groups via two very different derivations. In this paper, co-authored by researchers from both groups, we present these derivations: as a sequence of calls to PTS and as a non-parametric greedy variant of Anytime Repairing A ⁎. We describe experiments that evaluate the new algorithms in the 15-puzzle, KPP-COM, robot motion planning, gridworld navigation, and multiple sequence alignment search domains. Our results suggest that when compared with previous anytime algorithms, APTS/ANA ⁎: (1) does not require user-set parameters, (2) finds an initial solution faster, (3) spends less time between solution improvements, (4) decreases the suboptimality bound of the current-best solution more gradually, and (5) converges faster to an optimal solution when reachable.

ICRA Conference 2013 Conference Paper

Cloud-based robot grasping with the google object recognition engine

• Ben Kehoe
• Akihiro Matsukawa
• Sal Candido
• James J. Kuffner
• Ken Goldberg

Rapidly expanding internet resources and wireless networking have potential to liberate robots and automation systems from limited onboard computation, memory, and software. “Cloud Robotics” describes an approach that recognizes the wide availability of networking and incorporates open-source elements to greatly extend earlier concepts of “Online Robots” and “Networked Robots”. In this paper we consider how cloud-based data and computation can facilitate 3D robot grasping. We present a system architecture, implemented prototype, and initial experimental data for a cloud-based robot grasping system that incorporates a Willow Garage PR2 robot with onboard color and depth cameras, Google's proprietary object recognition engine, the Point Cloud Library (PCL) for pose estimation, Columbia University's GraspIt! toolkit and OpenRAVE for 3D grasping and our prior approach to sampling-based grasp analysis to address uncertainty in pose. We report data from experiments in recognition (a recall rate of 80% for the objects in our test set), pose estimation (failure rate under 14%), and grasping (failure rate under 23%) and initial results on recall and false positives in larger data sets using confidence measures.

IROS Conference 2013 Conference Paper

Sigma hulls for Gaussian belief space planning for imprecise articulated robots amid obstacles

• Alex X. Lee
• Yan Duan
• Sachin Patil
• John Schulman
• Zoe McCarthy
• Jur van den Berg
• Ken Goldberg
• Pieter Abbeel

In many home and service applications, an emerging class of articulated robots such as the Raven and Baxter trade off precision in actuation and sensing to reduce costs and to reduce the potential for injury to humans in their workspaces. For planning and control of such robots, planning in belief ssigma hullpace, i. e. , modeling such problems as POMDPs, has shown great promise but existing belief space planning methods have primarily been applied to cases where robots can be approximated as points or spheres. In this paper, we extend the belief space framework to treat articulated robots where the linkage can be decomposed into convex components. To allow planning and collision avoidance in Gaussian belief spaces, we introduce the concept of sigma hulls: convex hulls of robot links transformed according to the sigma standard deviation boundary points generated by the Unscented Kalman filter (UKF). We characterize the signed distances between sigma hulls and obstacles in the workspace to formulate efficient collision avoidance constraints compatible with the Gilbert-Johnson-Keerthi (GKJ) and Expanding Polytope Algorithms (EPA) within an optimization-based planning framework. We report results in simulation for planning motions for a 4-DOF planar robot and a 7-DOF articulated robot with imprecise actuation and inaccurate sensors. These experiments suggest that the sigma hull framework can significantly reduce the probability of collision and is computationally efficient enough to permit iterative re-planning for model predictive control.

ICRA Conference 2012 Conference Paper

A robot path planning framework that learns from experience

• Dmitry Berenson
• Pieter Abbeel
• Ken Goldberg

We propose a framework, called Lightning, for planning paths in high-dimensional spaces that is able to learn from experience, with the aim of reducing computation time. This framework is intended for manipulation tasks that arise in applications ranging from domestic assistance to robot-assisted surgery. Our framework consists of two main modules, which run in parallel: a planning-from-scratch module, and a module that retrieves and repairs paths stored in a path library. After a path is generated for a new query, a library manager decides whether to store the path based on computation time and the generated path's similarity to the retrieved path. To retrieve an appropriate path from the library we use two heuristics that exploit two key aspects of the problem: (i) A correlation between the amount a path violates constraints and the amount of time needed to repair that path, and (ii) the implicit division of constraints into those that vary across environments in which the robot operates and those that do not. We evaluated an implementation of the framework on several tasks for the PR2 mobile manipulator and a minimally-invasive surgery robot in simulation. We found that the retrieve-and-repair module produced paths faster than planning-from-scratch in over 90% of test cases for the PR2 and in 58% of test cases for the minimally-invasive surgery robot.

ICRA Conference 2012 Conference Paper

Toward cloud-based grasping with uncertainty in shape: Estimating lower bounds on achieving force closure with zero-slip push grasps

• Ben Kehoe
• Dmitry Berenson
• Ken Goldberg

This paper explores how Cloud Computing can facilitate grasping with shape uncertainty. We consider the most common robot gripper: a pair of thin parallel jaws, and a class of objects that can be modeled as extruded polygons. We model a conservative class of push-grasps that can enhance object alignment. The grasp planning algorithm takes as input an approximate object outline and Gaussian uncertainty around each vertex and center of mass. We define a grasp quality metric based on a lower bound on the probability of achieving force closure. We present a highly-parallelizable algorithm to compute this metric using Monte Carlo sampling. The algorithm uses Coulomb frictional grasp mechanics and a fast geometric test for conservative conditions for force closure. We run the algorithm on a set of sample shapes and compare the grasps with those from a planner that does not model shape uncertainty. We report computation times with single and multi-core computers and sensitivity analysis on algorithm parameters. We also describe physical grasp experiments using the Willow Garage PR2 robot.

AAAI Conference 2011 Conference Paper

Anytime Nonparametric A*

• Jur van den Berg
• Rajat Shah
• Arthur Huang
• Ken Goldberg

Anytime variants of Dijkstra’s and A* shortest path algorithms quickly produce a suboptimal solution and then improve it over time. For example, ARA* introduces a weighting value (ε) to rapidly find an initial suboptimal path and then reduces ε to improve path quality over time. In ARA*, ε is based on a linear trajectory with ad-hoc parameters chosen by each user. We propose a new Anytime A* algorithm, Anytime Nonparametric A* (ANA*), that does not require ad-hoc parameters, and adaptively reduces ε to expand the most promising node per iteration, adapting the greediness of the search as path quality improves. We prove that each node expanded by ANA* provides an upper bound on the suboptimality of the current-best solution. We evaluate the performance of ANA* with experiments in the domains of robot motion planning, gridworld planning, and multiple sequence alignment. The results suggest that ANA* is as efficient as ARA* and in most cases: (1) ANA* finds an initial solution faster, (2) ANA* spends less time between solution improvements, (3) ANA* decreases the suboptimality bound of the current-best solution more gradually, and (4) ANA* finds the optimal solution faster. ANA* is freely available from Maxim Likhachev’s Search-based Planning Library (SBPL).

IROS Conference 2011 Conference Paper

EG-RRT: Environment-guided random trees for kinodynamic motion planning with uncertainty and obstacles

• Leonard Jaillet
• Judy Hoffman
• Jur van den Berg
• Pieter Abbeel
• Josep M. Porta
• Ken Goldberg

Existing sampling-based robot motion planning methods are often inefficient at finding trajectories for kinodynamic systems, especially in the presence of narrow passages between obstacles and uncertainty in control and sensing. To address this, we propose EG-RRT, an Environment-Guided variant of RRT designed for kinodynamic robot systems that combines elements from several prior approaches and may incorporate a cost model based on the LQG-MP framework to estimate the probability of collision under uncertainty in control and sensing. We compare the performance of EG-RRT with several prior approaches on challenging sample problems. Results suggest that EG-RRT offers significant improvements in performance.

ICRA Conference 2010 Conference Paper

Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations

• Jur van den Berg
• Stephen Miller
• Daniel Duckworth
• Humphrey Hu
• Andrew Wan
• Xiao-Yu Fu
• Ken Goldberg
• Pieter Abbeel

In the future, robotic surgical assistants may assist surgeons by performing specific subtasks such as retraction and suturing to reduce surgeon tedium and reduce the duration of some operations. We propose an apprenticeship learning approach that has potential to allow robotic surgical assistants to autonomously execute specific trajectories with superhuman performance in terms of speed and smoothness. In the first step, we record a set of trajectories using human-guided backdriven motions of the robot. These are then analyzed to extract a smooth reference trajectory, which we execute at gradually increasing speeds using a variant of iterative learning control. We evaluate this approach on two representative tasks using the Berkeley Surgical Robots: a figure eight trajectory and a two handed knot-tie, a tedious suturing sub-task required in many surgical procedures. Results suggest that the approach enables (i) rapid learning of trajectories, (ii) smoother trajectories than the human-guided trajectories, and (iii) trajectories that are 7 to 10 times faster than the best human-guided trajectories.

ICRA Conference 2009 Conference Paper

Guiding medical needles using single-point tissue manipulation

• Meysam Torabi
• Kris Hauser
• Ron Alterovitz
• Vincent Duindam
• Ken Goldberg

This paper addresses the use of robotic tissue manipulation in medical needle insertion procedures to improve targeting accuracy and to help avoid damaging sensitive tissues. To control these multiple, potentially competing objectives, we present a phased controller that operates one manipulator at a time using closed-loop imaging feedback. We present an automated procedure planning technique that uses tissue geometry to select the needle insertion location, manipulation locations, and controller parameters. The planner uses a stochastic optimization of a cost function that includes tissue stress and robustness to disturbances. We demonstrate the system on 2D tissues simulated with a mass-spring model, including a simulation of a prostate brachytherapy procedure. It can reduce targeting errors from more than 2 cm to less than 1 mm, and can also shift obstacles by over 1 cm to clear them away from the needle path.

IROS Conference 2009 Conference Paper

Nonparametric belief propagation for distributed tracking of robot networks with noisy inter-distance measurements

• Jeremy Schiff
• Erik B. Sudderth
• Ken Goldberg

We consider the problem of tracking multiple moving robots using noisy sensing of inter-robot and inter-beacon distances. Sensing is local: there are three fixed beacons at known locations, so distance and position estimates propagate across multiple robots. We show that the technique of Nonparametric Belief Propagation (NBP), a graph-based generalization of particle filtering, can address this problem and model multi-modal and ring-shaped uncertainty distributions. NBP provides the basis for distributed algorithms in which messages are exchanged between local neighbors. Generalizing previous approaches to localization in static sensor networks, we improve efficiency and accuracy by using a dynamics model for temporal tracking. We compare the NBP dynamic tracking algorithm with SMCL+R, a sequential Monte Carlo algorithm. Whereas NBP currently requires more computation, it converges in more cases and provides estimates that are 3 to 4 times more accurate. NBP also facilitates probabilistic models of sensor accuracy and network connectivity.

IROS Conference 2009 Conference Paper

Planning fireworks trajectories for steerable medical needles to reduce patient trauma

• Jijie Xu
• Vincent Duindam
• Ron Alterovitz
• Jean Pouliot
• J. Adam M. Cunha
• I-Chow Hsu
• Ken Goldberg

Accurate insertion of needles to targets in 3D anatomy is required for numerous medical procedures. To reduce patient trauma, a ¿fireworks¿ needle insertion approach can be used in which multiple needles are inserted from a single small region on the patient's skin to multiple targets in the tissue. In this paper, we explore motion planning for ¿fireworks¿ needle insertion in 3D environments by developing an algorithm based on Rapidly-exploring Random Trees (RRTs). Given a set of targets, we propose an algorithm to quickly explore the configuration space by building a forest of RRTs and to find feasible plans for multiple steerable needles from a single entry region. We present two path selection algorithms with different optimality considerations to optimize the final plan among all feasible outputs. Finally, we demonstrate the performance of the proposed algorithm with a simulation based on a prostate cancer treatment environment.

IROS Conference 2009 Conference Paper

Surgical retraction of non-uniform deformable layers of tissue: 2D robot grasping and path planning

• Rik Jansen
• Kris Hauser
• Nuttapong Chentanez
• A. Frank van der Stappen
• Ken Goldberg

This paper considers robotic automation of a common surgical retraction primitive of exposing an underlying area by grasping and lifting a thin, 3D, possibly inhomogeneous layer of tissue. We present an algorithm that computes a set of stable and secure grasp-and-retract trajectories for a point-jaw gripper moving along a plane, and runs a 3D finite element (FEM) simulation to certify and assess the quality of each trajectory. To compute secure candidate grasp locations, we use a continuous spring model of thin, inhomogeneous deformable objects with linear energy potential. Experiments show that this method produces many of the same grasps as an exhaustive optimization with an FEM mesh, but is orders of magnitude cheaper: our method runs in O(v log v) time, where v is the number of veins, while the FEM computation takes O(pn 3 ) time, where n is the number of nodes in the FEM mesh and p is the number of nodes on its perimeter. Furthermore, we present a constant tissue curvature (CTC) retraction trajectory that distributes strain uniformly around the medial axis of the tissue. 3D FEM simulations show that the CTC achieves retractions with lower tissue strain than circular and linear trajectories. Overall, our algorithm computes and certifies a high-quality retraction in about one minute on a PC.

ICRA Conference 2008 Conference Paper

Screw-based motion planning for bevel-tip flexible needles in 3D environments with obstacles

• Vincent Duindam
• Ron Alterovitz
• S. Shankar Sastry
• Ken Goldberg

Bevel-tip flexible needles have greater mobility than straight rigid needles, and can be used to reach targets behind sensitive or impenetrable areas. Accurately planning and executing the optimal motions for such steerable needles is difficult, however, and requires solving inverse kinematics for a nonholonomic system. This paper presents an approach to 3D motion planning for bevel-tip needles in an environment with obstacles. Instead of discretizing the configuration space as in earlier work, we discretize the control space, such that the trajectory of the needle can be expressed analytically without the need for approximate numerical simulation. This results in a fast optimization routine that finds a locally optimal path in a 3D environment with obstacles, requiring just a few seconds of computation time on a standard PC. We introduce two different discretization strategies that lead to differently structured paths and show that both produce valid trajectories from start to goal. To our knowledge, the presented method is the first to address motion planning for bevel-tip needles in a 3D environment with obstacles.

IROS Conference 2007 Conference Paper

Respectful cameras: detecting visual markers in real-time to address privacy concerns

• Jeremy Schiff
• Marci Meingast
• Deirdre K. Mulligan
• S. Shankar Sastry
• Ken Goldberg

To address privacy concerns with digital video surveillance cameras, we propose a practical, real-time approach that preserves the ability to observe actions while obscuring individual identities. In our proposed respectful cameras system, people who wish to remain anonymous agree to wear colored markers such as a hat or vest. The system automatically tracks these markers using statistical learning and classification to infer the location and size of each face and then inserts elliptical overlays. Our objective is to obscure the face of each individual wearing a marker, while minimizing the overlay area in order to maximize the remaining observable region of the scene. Our approach incorporates a visual color-tracker based on a 9 dimensional color-space by using a probabilistic AdaBoost classifier with axis-aligned hyperplanes as weak-learners. We then use particle filtering to incorporate interframe temporal information. We present experiments illustrating the performance of our system in both indoor and outdoor settings, where occlusions, multiple crossing targets, and lighting changes occur. Results suggest that the respectful camera system can reduce false negative rates to acceptable levels (under 2%).

ICRA Conference 2006 Conference Paper

A Minimum Variance Calibration Algorithm for Pan-tilt Robotic Cameras in Natural Environments

• Dezhen Song
• Ni Qin
• Ken Goldberg

A new generation of inexpensive robotic pan-tilt cameras can maintain high-resolution panoramic displays of natural environments. However, the pan-tilt mechanisms are imprecise: small errors can produce large errors in the panoramic display. It is thus important to accurately estimate pan-tilt values. We present a new calibration algorithm that does not rely on calibration markers or fixed orthogonal edges which are rarely available in natural scenes. Our calibration algorithm uses image variance density to optimally estimate camera pan and tilt values by incrementally refining image registration using overlapping images from prior frames. Experiments suggest that the new calibration algorithm can reduce calibration error by 81%. In a companion paper, we present a new image registration algorithm based on spherical projection that optimally aligns the resulting frames

ICRA Conference 2006 Conference Paper

Aligning Windows of Live Video From an Imprecise Pan-tilt-zoom Robotic Camera into a Remote Panoramic Display

• Ni Qin
• Dezhen Song
• Ken Goldberg

A pan-tilt-zoom robotic camera can provide detailed live video of selected areas of interest within a large potential viewing field. To provide spatial context for human observers, it is desirable to insert the resulting live video into a large spherical panoramic display representing the entire viewing field. Accurate alignment of the video stream within the panoramic display is difficult due to small errors in the robot pan-tilt values and image distortion due to nonlinear projection. Existing image alignment algorithms cannot keep up with rapid changes in camera position. In this paper, we present a constant-time image alignment algorithm based on spherical projection and projection-invariant selective sampling that accurately registers paired images at 25 frames per second on a standard PC. Experiments suggest that the new alignment algorithm is faster than previous algorithms by a factor four or more. In a companion paper, we present a new calibration algorithm based on image variance density that optimally estimates camera pan-tilt parameters

ICRA Conference 2006 Conference Paper

Blades: a New Class of Geometric Primitives for Feeding 3D Parts on Vibratory Tracks

• Onno C. Goemans
• Ken Goldberg
• A. Frank van der Stappen

The vibratory bowl feeder remains the most common approach to the automated feeding (orienting) of industrial parts. We study the algorithmic design of devices on the bowl feeder track that filter out all but one orientation of a given polyhedral part. In this context, we propose a simple new primitive, consisting of one horizontally mounted convex polygonal metal "blade", that can feed a broad class of three-dimensional parts by reorienting and rejecting all but those in a desired orientation. This powerful new 3D geometric feeding primitive combines the reorientation functionality of fences with the rejection functionality of traps. Due to its simplicity, the proposed primitive allows for the development of methods to automate its design process. We present a complete procedure that takes as input any polyhedral part along with its center of mass. Given this input, the procedure identifies all single blade solutions that feed the part. The output is either the set of all valid blade designs or a notification that the part cannot be fed using a single blade

ICRA Conference 2005 Conference Paper

Planning for Steerable Bevel-tip Needle Insertion Through 2D Soft Tissue with Obstacles

• Ron Alterovitz
• Ken Goldberg
• Allison M. Okamura

We explore motion planning for a new class of highly flexible bevel-tip medical needles that can be steered to previously unreachable targets in soft tissue. Planning for these procedures is difficult because the needles bend during insertion and cause the surrounding soft tissues to displace and deform. In this paper, we develop a planning algorithm for insertion of highly flexible bevel-tip needles into soft tissues with obstacles in a 2D imaging plane. Given an initial needle insertion plan specifying location, orientation, bevel rotation, and insertion distance, the planner combines soft tissue modeling and numerical optimization to generate a needle insertion plan that compensates for simulated tissue de formations, locally avoids polygonal obstacles, and minimizes needle insertion distance. The simulator computes soft tissue deformations using a finite element model that incorporates the effects of needle tip and frictional forces using a 2D mesh. We formulate the planning problem as a constrained nonlinear optimization problem that is locally minimized using a penalty method that converts the formulation to a sequence of unconstrained optimization problems. We apply the planner to bevel-right and bevel-left needles and generate plans for targets that are unreachable by rigid needles.

IROS Conference 2005 Conference Paper

Steering flexible needles under Markov motion uncertainty

• Ron Alterovitz
• Andrew E. B. Lim
• Ken Goldberg
• Gregory S. Chirikjian
• Allison M. Okamura

When inserted into soft tissues, flexible needles with bevel tips have been shown experimentally to follow a path of constant curvature in the direction of the bevel. By controlling 2 degrees of freedom at the needle base (bevel direction and insertion distance), these needles can be steered around obstacles to reach targets inaccessible to rigid needles. Motion planning for needle steering is a type of nonholonomic planning for a Dubins car with no reversal. We develop a motion planning algorithm based on dynamic programming where the path of the needle is uncertain due to uncertainty in tissue properties, needle mechanics, and interaction forces. The algorithm computes a discrete control sequence of insertions and direction changes so the needle reaches a target in an imaging plane while minimizing expected cost due to insertion distance, direction changes, and obstacle collisions. We efficiently sample the state space of needle tip positions and orientations and define bounds on the errors due to discretization. We formulate the motion planning problem as a Markov decision process (MDP) and use infinite horizon dynamic programming to compute an optimal control sequence. We first apply the method to the deterministic motion case where the needle precisely follows a path of constant curvature and then to the uncertain motion case where state transitions are defined by a probability distribution. Our implementation generates motion plans for bevel-tip needles that reach targets inaccessible to rigid needles and demonstrates that accounting for uncertainty can lead to significantly different motion plans.

ICRA Conference 2004 Conference Paper

An Exact Algorithm Optimizing Coverage-resolution for Automated Satellite Frame Selection

• Dezhen Song
• A. Frank van der Stappen
• Ken Goldberg

Near real time satellite imaging provides timely images of the earth for weather prediction, disaster response, search and rescue, surveillance, and defense applications. As the satellite passes over the earth, camera imaging parameters are changed during each time window based on demand for images, specified as user requested zones in the reachable field of view during that time window. The satellite frame selection (SFS) problem is to find the camera frame parameters that maximize reward during each time window. To automate satellite management, we formalize the SFS problem based on a new reward metric that incorporates both image resolution and coverage. For a set of n client requests we give a series of algorithms, the fastest computes optimal results in O(n/sup 3/) for satellites with continuously variable resolution. We have implemented the algorithms and compare computation speed for all algorithms.

ICRA Conference 2004 Conference Paper

D-space and Deform Closure: a Framework for Holding Deformable Parts

• K. Gopalakrishnan 0002
• Ken Goldberg

We extend the form closure framework for rigid parts to holding a class of deformable parts. In this class, a part is a linearly elastic, frictionless polygon with a finite element mesh and given stiffness matrix. We define the D-space (deformation-space) of a part as the C-space of all its mesh nodes. We define the free space as the intersection of the set of topology preserving mesh configurations with the complement of the union of all D-obstacles that represent collisions of the part with finger bodies. Consider a given set of finger bodies in frictionless contact with a part. When positive work is needed to release the part, we say that it is in deform closure. We present a numerical example and prove two results: (1) if contact set holds a rigid part in form closure, it will hold the equivalent deformable part in deform closure; and (2) deform closure is frame invariant.

ICRA Conference 2004 Conference Paper

Unsupervised Scoring for Scalable Internet-based Collaborative Teleoperation

• Ken Goldberg
• Dezhen Song
• In Yong Song
• Jane McGonigal
• Wei Zheng
• Dana Plautz

For applications in education and entertainment, scalable Internet-based collaborative teleoperation allows many users simultaneously to share control of a single device. Automated numerical methods that can assess and record performance provide an incentive for users to participate and a means to evaluate individual and group performance. In this paper we describe "unsupervised scoring": a numerical approach to assessment based on clustering and response time. Like unsupervised learning, this approach is based on identifying regularities in the input rather than comparing input with desired output specified by an external supervisor. We present an algorithm for rapidly computing user scores that scales linearly with the number of users. We describe an implemented Java-based user interface incorporating this metric, an application based on the classic Twister game, and results where individual scores are compared with group performance.

ICRA Conference 2003 Conference Paper

"Unilateral" fixturing of sheet metal parts using modular jaws with plane-cone contacts

• K. Gopalakrishnan 0002
• Matthew Zaluzec
• Rama Koganti
• Patricia Deneszczuk
• Ken Goldberg

To fixture sheet metal parts for welding, we propose "unilateral fixtures" consisting of modular fixturing elements that lie almost completely on one side of the part. These are based on cylindrical jaws with conical grooves which provide the equivalent of 4 point contacts. We propose a two-phase procedure for designing unilateral fixtures. The first phase is a geometric algorithm that assumes the part is rigid and computes vg-grips (vertex-groove grips). The vg-grip algorithm uses a fast sufficient test for immobility to generate a list of vg-grips and find bounds on jaw cone angles for each. The second phase is a fast heuristic procedure that uses FEM to arrange secondary contacts to reduce part deformation. For a part described by n concave "virtual vertices", a list of vg-grips and minimum half cone angles for each vg-grip can be generated in O(n/sup 2/) time. We also propose a quality metric based on the sensitivity of the part's orientation to an infinitesimal relaxation of the jaws that can be evaluated in constant time for a given fixture. For an FEM model with m nodes, the second phase takes O(m/sup 3/r) time to arrange r secondary contacts for each vg-grip.

ICRA Conference 2003 Conference Paper

Needle insertion and radioactive seed implantation in human tissues: simulation and sensitivity analysis

• Ron Alterovitz
• Ken Goldberg
• Jean Pouliot
• Richard Tascherau
• I-Chow Hsu

To facilitate training and planning for medical procedures such as prostate brachytherapy, we are developing an interactive simulation of needle insertion and radioactive seed implantation in soft tissues. We describe a new 2D dynamic FEM model based on a reduced set of scalar parameters such as needle friction, sharpness, and velocity, where the mesh is updated to maintain element boundaries along the needle shaft and the effects of needle tip and frictional forces are simulated. The computational complexity of our model grows linearly with the number of elements in the mesh and achieves 24 frames per second for 1250 triangular elements on a 750 MHz PC. We use the simulator to characterize the sensitivity of seed placement error to physician-controlled and biological parameters. Results indicate that seed placement error is highly sensitive to physician-controlled parameters such as needle position, sharpness, and friction, and less sensitive to patient-specific parameters such as tissue stiffness and compressibility.

IROS Conference 2003 Conference Paper

Sensorless planning for medical needle insertion procedures

• Ron Alterovitz
• Ken Goldberg
• Jean Pouliot
• Richard Tascherau
• I-Chow Hsu

Medical procedures such as seed implantation, biopsies, and treatment injections require inserting a needle tip to a specific target location inside the human body. This is difficult because (1) needle insertion causes soft tissues to displace and deform, and (2) it is often difficult or impossible to obtain precise imaging data during insertion. We are developing a sensorless planning system for needle insertion that incorporates numerical optimization with a soft tissue simulation based on a dynamic FEM formulation that models the effects of needle tip and frictional forces using a 2D mesh. In this paper we describe a sensorless planning algorithm for radioactive seed implantation that computes needle insertion offsets that compensate for tissue deformations. We apply the method to seed implantation during permanent seed prostate brachytherapy to minimize seed placement error in simulation without relying on real-time imaging.

IROS Conference 2003 Conference Paper

ShareCam part 1: interface, system architecture, and implementation of a collaboratively controlled robotic Webcam

• Dezhen Song
• Ken Goldberg

ShareCam is a robotic pan, tilt, and zoom web-based camera controlled by simultaneous frame requests from online users. Part II describes algorithms. This paper, part I, focuses on the system. Robotic Webcameras are commercially available but currently restrict control only one user at a time. ShareCam introduces a new interface that allows simultaneous control many users. In this Java-based interface, participating users interact desired frames remotely located browsers where users draw desired frames over a fixed panoramic image. User inputs re transmitted back to pair of PC servers that compute optimal camera back to a pair of PC servers that compute optimal camera parameters, servo the camera, and provide a video stream to all users. We describe the system, online experiments, and compare results with two frame selection models based on user "satisfaction", one memoryless and the second based on satisfaction over multiple motion cycles.

IROS Conference 2003 Conference Paper

ShareCam part II: approximate and distributed algorithms for a collaboratively controlled robotic Webcam

• Dezhen Song
• Anatol Pashkevich
• Ken Goldberg

ShareCam is a robotic pan, tilt, and zoom Web-based camera controlled by simultaneous frame requests from online users. Part I describes the system. This paper, part II, focuses on algorithms. The ShareCam problem is to find a camera frame that optimizes a measure of total user satisfaction. We present a grid-based approximation algorithm: given camera frame requests from n users, and approximation bound /spl epsi/, we analyze the trade of between solution quality and processing speed and prove that the algorithm runs in O(n//spl epsi//sup 3/) time. The algorithm can be distributed to run in O(1//spl epsi//sup 3/) time at each client and in O(n + 1//spl epsi//sup 3/) time at the server. Experiments suggest that performance of the distributed algorithm degrades gracefully as clients fail to complete their part of the computation.

ICRA Conference 2002 Conference Paper

Collaborative Online Teleoperation with Spatial Dynamic Voting and a Human "Tele-Actor"

• Ken Goldberg
• Dezhen Song
• Yoek-Nam Khor
• David Pescovitz
• Anthony Levandowski
• Jesse C. Himmelstein
• Janice Shih
• Annamarie Ho

Internet-based "online robots" now provide public access to remote locations such as museums and laboratories. The Tele-Actor is a collaborative online teleoperation system for distance learning that allows many students to simultaneously share control of a single mobile resource. Our goal is to preserve the educational advantages of field trips without the drawbacks of group travel. We propose the "spatial dynamic voting" (SDV) interface for multiple operator single robot (MOSR) teleoperation. The SDV collects, displays, and analyzes a sequence of spatial votes from multiple online operators at their Internet browsers. The votes drive the motion of a single mobile robot or human "Tele-Actor". The paper describes Version 3. 0 of the system architecture, SDV interface, algorithms for automated goal selection, and metrics for collaboration and leadership. We report results from a July 2001 field test with 56 remote users.

ICRA Conference 2002 Conference Paper

Fixturing Hinged Polygons

• Jae-Sook Cheong
• Ken Goldberg
• Mark H. Overmars
• A. Frank van der Stappen

We study the problem of fixturing a chain of hinged objects in a given placement with frictionless point contacts. We define the notions of immobility and robust immobility - which are comparable to the second and first order immobility for a single object - to capture the intuitive requirement for the fixture of a chain of hinged objects. Robust immobility differs from immobility in that it additionally requires insensitivity to small perturbations of contacts. We show that (p+2) frictionless point contacts can immobilize any chain of p/spl ne/3 polygons without parallel edges; six contacts can immobilize any chain of three such polygons. Any chain of p arbitrary polygons can be immobilized with at most (p+4) contacts. We also show that /spl lceil/(6/5)(p+2)/spl rceil/ contacts suffice to robustly immobilize p polygons without parallel edges, and that /spl lceil/(5/4)(p+2)/spl rceil/ contacts can robustly immobilize p/spl ne/3 arbitrary polygons, and eight contacts can robustly immobilize three polygons.

ICRA Conference 2002 Conference Paper

Gripping Parts at Concave Vertices

• K. Gopalakrishnan 0002
• Ken Goldberg

A simple gripper with two vertical cylindrical jaws can make contact with external or internal concavities in polygonal and polyhedral parts to align and grip parts in form closure. This is called a /spl nu/-grip. We begin by defining 2D /spl nu/-grips, where a pair of frictionless point jaws makes contact with a pair of polygonal part concavities to achieve form-closure. We define a /spl nu/-grip quality metric based on the maximum possible change in the part's orientation when jaw position is relaxed infinitesimally. For a polygonal part with polygonal holes, we give an algorithm for computing and ranking 2D /spl nu/-grips. We also extend the definition to jaws with non-zero radii. In 3D, /spl nu/-grips are achieved with a pair of frictionless vertical cylinders. We define 3D /spl nu/-grips and give a numerical algorithm for computing all 3D /spl nu/-grips of a polyhedral part. If n is the number of vertices that describe the part and k is the number of concave vertices, we can compute all 2D /spl nu/-grips in O(n+k/sup 2/) time. Measures of complexity are given for computing offsets for jaws with nonzero radii, a ranked list of 2D /spl nu/-grips based on the quality metric, and all 3D /spl nu/-grips. A Java implementation of the 2D algorithm is available.

ICRA Conference 2002 Conference Paper

Orienting Micro-Scale Parts with Squeeze and Roll Primitives

• Mark Moll
• Ken Goldberg
• Michael A. Erdmann
• Ronald S. Fearing

Orienting parts that measure only a few micrometers in diameter introduces several challenges that need not be considered at the macro-scale. First, there are several kinds of sticking effects due to Van der Waals forces and static electricity which complicate hand-off motions and release of a part. Second, the degrees of freedom of micromanipulators are limited. The paper proposes a pair of manipulation primitives and a complete algorithm that addresses these challenges. We show that a sequence of these two manipulation primitives can uniquely orient any asymmetric part while maintaining contact without sensing. This allows us to apply the same plan to many (identical) parts simultaneously. For asymmetric parts we can find a plan of length O(n) in O(n) time that orients the part, where n is the number of vertices.

IROS Conference 2001 Conference Paper

Collaborative control of robot motion: robustness to error

• Ken Goldberg
• Billy Chen

We consider collaborative control systems, where multiple sources share control of a single robot. These sources could come from multiple sensors (sensor fusion), multiple control processes (subsumption), or multiple human operators. Reports suggest that such systems are highly fault tolerant, even with large numbers of sources. We develop a formal model, modeling sources with finite automata. A collaborative ensemble of sources generates a single stream of incremental steps to control the motion of a point robot moving in the plane. We first analyze system performance with a uniform ensemble of well-behaved deterministic sources. We then model malfunctioning sources that go silent or generate inverted control signals. We discover that performance initially improves in the presence of malfunctioning sources and remains robust even when a sizeable fraction of sources malfunction. Initial tests suggest similar results with non-deterministic (random) sources. The formal model may also provide insight into how humans can share control of an online robot.

ICRA Conference 2001 Conference Paper

Design of Robot Gripper Jaws based on Trapezoidal Modules

• Mike Tao Zhang
• Ken Goldberg

Assembly line reliability can be increased if grippers are carefully designed to capture and align parts. Jaw design depends on the geometric and mechanical properties of the part as well as its desired orientation. We propose a class of modular jaws based on rapidly machineable trapezoidal modules. An optimal jaw design is an arrangement of trapezoidal jaw modules that maximizes contacts between the gripper and the part at its desired final orientation over the constraints that: the jaws will capture and rotate the part to its desired orientation and achieve a form-closure grasp. Given the n-sided 2D convex projection of an extruded polygonal part, we develop an implemented O(n/sup 5/) algorithm to find the optimal jaw design. The algorithm combines toppling, jamming, liftoff, accessibility, and form-closure analysis. We verify resulting designs by physical experiments.

ICRA Conference 2001 Conference Paper

Orienting Parts by Inside-out Pulling

• Robert-Paul Berretty
• Ken Goldberg
• Mark H. Overmars
• A. Frank van der Stappen

A common task in automated manufacturing processes is that of orienting (or feeding) parts prior to assembly. We propose a new type of feeder. We consider sensorless orientation of polygonal parts with elevated edges by pull actions with an overhead finger. We show that any asymmetric convex polygonal part can be oriented by a sequence of pull operations. We give an O(n/sup 3/) algorithm to compute the shortest sequence of pull operations to orient a convex polygonal part with n vertices, if such a sequence exists. We also show that there exist non-convex parts that cannot be fed by a sequence of pull operations.

IROS Conference 2001 Conference Paper

Shape tolerance for robot gripper jaws

• Mike Tao Zhang
• Lawrence Cheung
• Ken Goldberg

In previous work (2001), we identified a class of robot grippers that can orient and grasp parts using an arrangement of trapezoidal jaw modules. In this paper, we define a parametric tolerance class for the jaws, such that the part alignment is guaranteed for all jaw geometry in the class. This tolerance class is derived based on the analysis of toppling, motion trajectory, and form-closure. We describe the O(n log n) algorithm to compute the parametric tolerance class based on maximal and minimal jaw specifications. We implemented the algorithm and report the results of physical experiments.

ICRA Conference 2000 Conference Paper

Collaborative Teleoperation via the Internet

• Ken Goldberg
• Billy Chen
• Rory Solomon
• Steve Bui
• Bobak Farzin
• Jacob Heitler
• Derek Poon
• Gordon Smith

We describe a system that allows a distributed group of users to simultaneously teleoperate an industrial robot arm via the Internet. A Java applet at each client streams mouse motion vectors from up to 3D users; a sewer aggregates these inputs to produce a single control stream for the robot. Users receive visual feedback from a digital camera mounted above the robot arm. To our knowledge, this is the first collaboratively controlled robot on the Internet.

ICRA Conference 2000 Conference Paper

The Toppling Graph: Designing Pin Sequences for Part Feeding

• Mike Tao Zhang
• Gordon Smith
• Robert-Paul Berretty
• Mark H. Overmars
• Ken Goldberg

We consider a sensorless approach to feeding parts on a conveyor belt using pins (rigid barriers) to topple parts into desired orientations. Given the 2D projection of an n-sided convex polyhedral part, its center of mass, and coefficients of friction, we give an O(n/sup 2/) algorithm to compute the "toppling graph", a new data structure that represents the mechanics of toppling, including rolling and jamming. The toppling graph can be used to identify critical pin heights that permit toppling. We compare pin heights predicted by the graph with physical experiments, and give a complete O(n/sup 3n/) algorithm for designing pin sequences.

ICRA Conference 1999 Conference Paper

Computing Parallel-Jaw Grips

• Gordon Smith
• Eric Lee
• Ken Goldberg
• Karl-Friedrich Böhringer
• John Craig

We propose a new model for gripping parts with an industrial parallel-jaw gripper. In contrast to many previous models, we use two grip points and define candidate grips that are resistant to slipping and torque about the part's center of mass. These grips must also be accessible and robust to perturbations in part position. We give an O(n/sup 3/) algorithm for computing and ranking O(n/sup 2/) such grips on an n-sided polygonal slice through the part. The algorithm will be part of a design and simulation system that can rapidly provide feedback to designers; thus it must run quickly and reliably. We have also implemented the algorithm in a Java applet with a graphical user interface that allows Internet users to define a part; the applet computes, ranks, and displays the set of computed grips.

ICRA Conference 1999 Conference Paper

Trap Design for Vibratory Bowl Feeders

• Robert-Paul Berretty
• Ken Goldberg
• Lawrence Cheung
• Mark H. Overmars
• Gordon Smith
• A. Frank van der Stappen

The vibratory bowl feeder is the oldest and still most common approach to the automated feeding (orienting) of industrial parts. We consider a class of vibratory bowl filters that can be described by removing polygonal sections from the track; we refer to this class of filters as traps. For an n-sided convex polygonal part and m-sided convex polygonal trap, we give an O((n+m)log(n+m)) algorithm to decide if the part will be rejected by the trap, and an O((nm(n+m))/sup 1+/spl epsiv//) algorithm which deals with non-convex parts and traps. We then consider the problem of designing traps for a given part, and consider two rectilinear subclasses, balconies and gaps. We give linear and O(n/sup 2/) algorithms for designing feeders and have tested the results with physical experiments using a commercial inline vibratory feeder.

IROS Conference 1998 Conference Paper

Estimating and optimizing throughput of a robotic part feeder using queueing theory

• Dadi Gudmundsson
• Ken Goldberg

We study a programmable robotic part feeder that relies on a sequence of three conveyor belts to singulate and re-circulate parts. In industrial practice, belt speeds are set in an ad-hoc fashion. Experience with real feeders reveals that throughput can suffer due to (1) starvation where no parts are visible to the camera and (2) saturation, where too many parts are visible, which prevents identifying part pose or grasping due to obstruction by nearby parts. This motivates our search for a systematic approach to setting belt speeds. This paper introduces models based on a 2D Poisson process for both intermittent and continuous motion feeding. For intermittent motion feeding we apply renewal theory to approximate and optimize the theoretical throughput. For continuous motion feeding we use a M/G/1 queue with customer impatience to approximate and optimize the theoretical throughput. We show that the analytic theory compares very well with simulation studies. For both models we show how to optimize the throughput when there is a constraint on the expected number of times a part should go through the system.

ICRA Conference 1998 Conference Paper

Parallel Microassembly with Electrostatic Force Fields

• Karl-Friedrich Böhringer
• Ken Goldberg
• Michael Cohn
• Roger Howe
• Albert P. Pisano

Microscopic (submillimeter) parts are often fabricated in parallel at high density but must then be assembled into patterns with lower spatial density. We propose a new approach to microassembly using: 1) ultrasonic vibration to eliminate friction and adhesion; and 2) electrostatic forces to position and align parts in parallel. We describe experiments on the dynamic and frictional properties of collections of microscopic parts under these conditions. We first demonstrate that ultrasonic vibration can be used to overcome adhesive forces; we also compare part behavior in air and vacuum. Next, we demonstrate that parts can be positioned and aligned using a combination of vibration and electrostatic forces. Finally, we demonstrate part sorting by size. Our goal is a systematic method for designing implementable planar force fields for microassembly based on part geometry.

IROS Conference 1997 Conference Paper

Analysis of part motion on a longitudinally vibrating plate

• Dan Reznik
• John F. Canny
• Ken Goldberg

We analyze the dynamics of part motion for a novel type of planar parts feeder consisting of a longitudinally vibrating flat plate and a part placed on its surface. For each vibration cycle, the plate's velocity is held positive (forward motion) for a longer time than it is held negative (backward motion). This type of asymmetric vibration combined with the nonlinear nature of Coulomb friction causes the part to accelerate along a straight line to a terminal velocity called the "feed rate". The average force exerted by the plate on the part is shown to be proportional to the latter's deviation from the feed rate. In other words, the part behaves as if it were immersed in a forward moving viscous fluid. Expressions for the feed rate and viscosity constants are derived with respect to various physical and control parameters. Rigid-body dynamic simulation results are shown to be in good agreement with the analysis.

ICRA Conference 1997 Conference Paper

Simplifying complex CAD geometry with conservative bounding contours

• Yan Zhuang
• Ken Goldberg
• Mary Pickett

Megabytes of data are often required for today's detailed CAD models. Users are often willing to sacrifice some of this detail for efficient rendering, transmission over networks, and robot motion planning. For example, many robotics and automation applications such as collision checking require only the simple bounding envelope of a complex set of nested CAD models. In contrast to approximate graphics rendering, these applications require that the simplified envelope be conservative: that the true CAD model be strictly contained inside it. One conservative bounding envelope is the bounding box. Another is the convex hull. Generally each conservative bounding envelope can be "trimmed" by a conservative bounding contour (CBC) of its 2D projection. This yields a hierarchy of conservative bounding envelopes of the original CAD model. In this paper we describe a method for generating the CBC at different levels of simplification. We give an algorithm that computes the CBC of a given 2D projection of a wireframe model. Since the wireframe requires neither boundary representation (BREP) nor constructive solid geometry (CSG) our method is compatible with almost all existing CAD standards.

ICRA Conference 1997 Conference Paper

Tuning robotic part feeder parameters to maximize throughput

• Dadi Gudmundsson
• Ken Goldberg

We study a programmable robotic part feeder that relies on a sequence of three conveyor belts to separate and re-circulate parts. In industrial practice, belt speeds are set in an ad-hoc fashion. Experience with real feeders reveals that throughput can suffer due to (1) starvation where no parts are visible to the camera and (2) saturation, where too many parts are visible, which prevents identifying part pose or grasping due to obstruction by nearby parts. This motivates our search for a systematic approach to setting belt speeds. Our goal is to optimize throughput, measured in terms of how many parts per second are delivered from the robotic feeder. We describe a planar model of the belts with a Poisson arrival process to stochastically model how belt speeds affect throughput. Initially, we study the finite case where N parts are placed into the feeder and re-circulated until they are all delivered by the robot. Our first insight is that the vision belt should be run at maximum achievable velocity. We run simulations to empirically determine optimal buffer belt velocity as a function of lot size. Finally, we develop a theoretical model for the case where N=/spl infin/ which approximates common usage where the buffer is replenished before it becomes empty. From this model, we derive the optimal buffer belt velocity and show that it produces throughput five times greater than that achieved with ad-hoc settings.

ICRA Conference 1996 Conference Paper

A complete algorithm for designing passive fences to orient parts

• Jeff Wiegley
• Ken Goldberg
• Michael A. Peshkin
• Mike Brokowski

Peshkin and Sanderson (1988) showed that parts can be aligned as they move on a conveyor belt against a passive sequence of fences. In this paper the authors describe the first complete algorithm to design such sequences for a given convex polygonal part. The algorithm is complete in the sense that it is guaranteed to find a design if one exists and to terminate with a negative report otherwise. Based on an exact breadth-first search of the design space, the algorithm is also guaranteed to find the design requiring the fewest fences. The authors describe the algorithm and compare results with those previously reported. The authors conjecture that a fence design exists to orient any convex polygonal part defined by a sequence of rational vertices.

ICRA Conference 1996 Conference Paper

Compiling assembly plans into hard automation

• Ken Goldberg
• Hadi Moradi

For high-volume assembly, "hard automation" systems consisting of a pipelined series of simple operations are often preferred over multi-axis robots for reasons of cost, speed, and precision. Given a desired final assembly, designing an appropriate assembly system requires the intuition of experienced designers who require up to six months to produce a design and may overlook a better design or fail to find a design when one exists. Thus there is a need for CAD tools that can assist the designers of assembly hardware and to give rapid feedback to product designers to facilitate design for assembly. We introduce a class of vertical assembly (VA) plans such that any valid VA plan can be "compiled" into a hard automation assembly system. VA plans consist of vertical insertions alternated with rotations such that parts and subassemblies at each stage sit stably on conveyor belts without fixtures. For an assembly with n vertices, we can decide if a VA plan exists in time O(n/sup 4/). We describe a complete and implemented planning algorithm that takes as input a 2D slice of an assembly consisting of polygonal parts and returns all valid VA plans ranked by a quality metric.

ICRA Conference 1996 Conference Paper

Estimating pose statistics for robotic part feeders

• Brian Mirtich
• Yan Zhuang
• Ken Goldberg
• John Craig
• Rob Zanutta
• Brian Carlisle
• John F. Canny

In automated assembly lines, part feeders often impose a bottleneck that restricts throughput. To facilitate the design of parts and assembly lines, the authors estimate feedrates based on CAD models of parts. A previous paper (Golberg and Craig, 1995) described how to predict throughput for a vision-based robotic part feeder given the distribution of part poses when parts are randomly dropped on a conveyor belt. Estimating this distribution is also useful for the design of traditional feeders such as vibratory bowls. In this paper the authors describe three algorithms for estimating pose distributions. The authors review the quasi-static estimate reported in Wiegley et al. (1992) and introduce a refinement that takes into account some measure of dynamic stability. The perturbed quasi-static estimate can be computed very rapidly and is more accurate than the quasi-static. Still more accurate are estimates based on Monte Carlo simulation using Impulse, although the latter comes at the penalty of increased computation time. The authors compare estimates from all three algorithms with physical experiments.

ICRA Conference 1995 Conference Paper

Complete Algorithms for Reorienting Polyhedral Parts Using a Pivoting Gripper

• Anil S. Rao
• David J. Kriegman
• Ken Goldberg

To rapidly feed industrial parts on an assembly line, Carlisle et. al. (1994) proposed a flexible part feeding system that drops parts on a flat conveyor belt, determines the pose of parts with a vision system and manipulates them into a desired pose. A robot arm with 4-DOF is capable of moving parts through 6-DOF when equipped with a passive pivoting axis between the parallel jaws of its gripper. We refer to these actions as pivot grasps. This paper considers the planning problem. Given a polyhedral part shape, coefficient of friction and a pair of stable configurations as input, find pairs of grasp points that will cause the part to pivot from one stable configuration to the other. For some transitions, pivot grasps may not exist. For a part with n faces and m stable configurations, we give an O(m/sup 2/n log n) algorithm to generate the m/spl times/m matrix of pivot grasps. When the part is star shaped, this reduces to O(m/sup 2/n). We also study a generalization that considers "capture regions" around stable configurations. Both algorithms are complete in that they are guaranteed to find pivot grasps when they exist.

ICRA Conference 1995 Conference Paper

Destop Teleoperation via the World Wide Web

• Ken Goldberg
• Michael Mascha
• Steve Gentner
• Nick Rothenberg
• Carl Sutter
• Jeff Wiegley

We built a system that allows a robot manipulator to be teleoperated via the WWW. Although the field of teleoperation dates back over 50 years, the WWW provides a low-cost and widely-available interface that can make teleoperated resources accessible to anyone with a desktop or laptop computer and modem. The "Mercury Project" consists of an industrial robot arm fitted with a CCD camera and a pneumatic system. We placed a sandbox filled with buried artifacts in the robot workspace. Using the ISMAP feature of HTTP, users can remotely move the camera to view desired locations or direct a short burst of compressed air into the sand to view the newly cleared region. To our knowledge, the Mercury Project is the first system to permit WWW users to remotely view and alter the real world. This paper focuses on interface design, robot hardware, and system architecture.

ICRA Conference 1995 Conference Paper

Sensorless manipulation using transverse vibrations of a plate

• Karl-Friedrich Böhringer
• Vivek Bhatt
• Ken Goldberg

The existing industrial parts feeders move the parts through a sequence of mechanical filters that reject parts in unwanted orientations. In this paper we develop a new setup that uses a different vibratory mechanism to systematically manipulate parts, by actively orienting and localizing them. The idea is to generate and change dynamic modes for a plate by varying the applied frequency of oscillation. Depending on the node shapes of the plate for these frequencies, the position and orientation of the parts can be controlled. We develop an analysis of the underlying dynamics, and show that it can be used to predict the behavior of objects placed on the vibrating plate. Using this analysis, we propose that the applied frequencies can be automatically sequenced to obtain a "sensorless" strategy for manipulating a given object.

ICRA Conference 1994 Conference Paper

"RISC" for Industrial Robotics: Recent Results and Open Problems

• John F. Canny
• Ken Goldberg

At the intersection of robotics, computational geometry, and manufacturing engineering, we have identified collection of research problems with near-term industrial applications. The common thread is robot systems with reduced intricacy in sensing and control (RISC), such as light beam sensors and parallel-jaw grippers. We conjecture that such systems, coupled with appropriate algorithms, are capable of recognizing and orienting a broad class of industrial parts. When compared with general-purpose robots, the resulting systems could be: (1) lower in cost, (2) more reliable, and (3) easier to reconfigure. The proposed hardware bears a close resemblance to existing "hard" automation; what is new is the application of computational methods for robust design and control of these systems, and more extensive use of simple sensors. By focusing on a small vocabulary of simple hardware, planning become computationally tractable and we can in some cases make guarantees about the existence of solutions. We borrowed the RISC acronym from computer architecture to acknowledge a common theme: identifying a minimal set of hardware primitives and matching these primitives with highly efficient software. In this paper, we review recent algorithms for locating, feeding, inserting and fixturing industrial parts. We discuss related work and propose a set of open problems for future research. >

ICRA Conference 1994 Conference Paper

A Complete Algorithm for Synthesizing Modular Fixtures for Polygonal Parts

• Randy C. Brost
• Ken Goldberg

Commercially-available modular fixturing systems typically include a square lattice of tapped and bushed holes with precise spacing and an assortment of precision locating and clamping elements that can be rigidly attached to the lattice using dowel pins or expanding mandrels. Currently, human expertise is required to synthesize a suitable arrangement of these elements to hold a given part. Besides being time consuming, if the set of alternatives is not systematically explored, the designer may fail to find an acceptable fixture or may settle upon a suboptimal fixture. The authors consider a class of modular fixtures that prevent a part from translating or rotating in the plane using four point contacts on the part's boundary. These fixtures are based on three round locators, each centered on a lattice point, and one translating clamp. The authors present an algorithm that accepts a polygonal part shape as input and synthesizes the set of all fixture designs that achieve form closure for the given part. The algorithm also allows the user to specify geometric access constraints on fixtures. If the part has n edges and its maximal diameter is d lattice units, the asymptotic running time of the algorithm is O(n/sup 5/d/sup 5/). The authors have implemented the algorithm and present example fixtures that it has synthesized. This implementation includes a metric to rank fixtures based on their ability to resist applied forces. The authors believe this is the first fixture synthesis algorithm that is complete in the sense that it is guaranteed to find an admissible fixture if one exists. Furthermore, the algorithm is guaranteed to find the optimal fixture, relative to any well-defined quality metric. >

ICRA Conference 1994 Conference Paper

A Pivoting Gripper for Feeding Industrial Parts

• Brian Carlisle
• Ken Goldberg
• Anil S. Rao
• Jeff Wiegley

To be cost effective and highly precise, many industrial assembly robots have only four degrees of freedom (DOF) plus a binary pneumatic gripper. Such robots commonly permit parts to be rotated only about a vertical axis. However it is often necessary to reorient parts about other axes prior to assembly. In this paper the authors describe a way to orient parts about an arbitrary axis by introducing a rotating bearing between the jaws of a simple gripper. Based on this mechanism, the authors are developing a rapidly configurable vision-based system for feeding parts. In this system, a camera determines initial part pose; the robot then reorients the part to achieve a desired final pose. The authors have implemented a prototype version in their laboratory using a commercially-available robot system. >

ICRA Conference 1994 Conference Paper

On the Existence of Modular Fixtures

• Yan Zhuang
• Ken Goldberg
• Yiu-Chung Wong

Modular fixtures are gaining wide use for flexible manufacturing and job shop machining. A modular fixture is an arrangement of fixture elements (fixels) that will locate and securely hold a given part. Typically, a human combines intuition with trial-and-error to design fixtures. In some cases designers are unable to design a fixture with given fixels and must resort to custom tooling. It is possible that human designers have overlooked a solution. It is also possible that no solution exists. In this paper the authors explore the existential question: given a fixture model and a part, does a fixture exist that will hold this part in form closure? If so, the authors say that the part is fixturable. The authors consider two classes of fixtures, one using 3 locators and a clamp, the other using 4 clamps. The authors provide one negative result-a class of cross-sections that is not fixturable-and two positive results-two classes of cross-sections that are guaranteed to be fixturable. These results give insight into the application range for different models of modular fixtures. >

ICRA Conference 1992 Conference Paper

A shape metric for design-for-assembly

• Gerard Jounghyun Kim
• George A. Bekey
• Ken Goldberg

The authors introduce a quantitative shape metric for planar parts, called feedability, based on a stochastic parts feeding algorithm reported by K. Goldberg et al. , (1990, 1991). Although feedability is only one possible metric for design-for-assembly (DFA), it relates one aspect of assembly cost directly to part geometry. Therefore, based on feedability, a set of geometric redesign strategies can be produced. The application of feedability to DFA is illustrated and compared with other DFA methods. >

ICRA Conference 1992 Conference Paper

Orienting generalized polygonal parts

• Anil S. Rao
• Ken Goldberg

K. Y. Goldberg (1990) described an algorithm for orienting planar polygonal parts using a modified parallel-jaw gripper. The authors extend this algorithm to handle parts with curved edges. They define a generalized polygon as a planar figure made up of piecewise linear and circular edges, and a generalized polygonal part as one of constant cross section, the convex hull of which is a generalized polygon. For this class of parts a complete algorithm for determining an optimal parts-orienting strategy is presented. >

IROS Conference 1992 Conference Paper

Shape Recognition By Random grasping

• Dukhyun Kang
• Ken Goldberg

ICRA Conference 1991 Conference Paper

Generating stochastic plans for a programmable parts feeder

• Ken Goldberg
• Matthew T. Mason
• Michael A. Erdmann

A programmable parts feeder, a mechanism that can be reprogrammed to handle differently shaped parts, is discussed. The authors present a planning algorithm that accepts an n-sided polygonal part as input and, in time O(n/sup 2/), generates a program (plan) for the feeder that maximizes expected feedrate. They have implemented the planner and verified some of the resulting plans in the laboratory. This work illustrates a stochastic framework for manipulation planning. >

ICRA Conference 1990 Conference Paper

Bayesian grasping

• Ken Goldberg
• Matthew T. Mason

A Bayesian approach to the problem of autonomous manipulation in the presence of state uncertainty is described. Uncertainty is modeled with a probability distribution on the state space. Each plan (sequence of actions) defines a mapping on the state space and hence a posterior probability distribution. An attempt is made to find a plan for optimizing expected performance. The Bayesian framework is applied to a grasping problem. A planar polygon whose initial orientation is described by a uniform distribution and a frictionless parallel-jaw gripper is assumed in order to plan automatically a sequence of open-loop squeezing operations to reduce orientational uncertainty and grasp the object. Although many different performance measures are possible depending on the application, the approach is illustrated by searching for plans that optimize the robot's expected throughput. >

ICRA Conference 1987 Conference Paper

Conditions for symmetric running in single- and double-support

• Ken Goldberg
• Marc H. Raibert

Even and odd functions of time - symmetric functions - have been used to simplify the control of running robots and have been observed in the body and leg motion of running animals. This paper explores the relationship between symmetric motion of the body and symmetric use of the legs in three planar models. The algebra of symmetric functions is applied to the equations of motion to show that symmetric leg actuation is required for symmetric body motion in the single-support case, but not in the double-support case unless additional constraints are imposed.