Author name cluster

Renaud Detry

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

17 papers

2 author rows

EWRL Workshop 2025 Workshop Paper

AREPO: Uncertainty-Aware Robot Ensemble Learning Under Extreme Partial Observability

Yurui Du
Louis Hanut
Herman Bruyninckx
Renaud Detry

Real-world applications of vision-based robot learning face two major challenges: extreme partial observability and effective simulation-to-reality (sim-to-real) transfer. This paper introduces a robust robot learning framework that enhances uncertainty awareness to address these challenges. We reinterpret variational-autoencoder–based visual reinforcement learning (RL) from an uncertainty-quantification perspective, enabling resilience to high sensory noise and severe visual occlusions—common in industrial robotic tasks. To further improve sim-to-real transfer, we propose an uncertainty-aware ensemble RL algorithm. We validate our methods on a laboratory task designed as a proxy for real-world industrial applications characterized by harsh environments with low visibility and physical occlusions. Both simulation and real-world results demonstrate significant improvements in task accuracy and efficiency over various baselines, highlighting the benefits of uncertainty-aware robot learning for complex operational contexts.

ICRA Conference 2025 Conference Paper

Robotic Framework for Iterative and Adaptive Profile Grading of Sand

Louis Hanut
Yurui Du
Andrew Vande Moere
Renaud Detry
Herman Bruyninckx

This paper studies sand profile grading, a manipulation task to obtain a desired geometric curve in sand. Manipulating sand is challenging because like other amorphous materials, its properties are difficult to estimate and emergent effects such as collapses may occur which both influence the manipulation outcome. To tackle these challenges, humans iterate and adapt their manual actions to the observed material states. In this paper, we propose to replicate this adaptive and iterative approach on a robotic profile grading task. Our results demonstrate that (1) tool insertion adaptation reduces force limit violations during tool-material interactions, (2) grading angle adaptation ensures no undercutting or collisions while allowing for cutting or smoothing the sand profile, and (3) adapting progress speed to task evolution provides a balance between grading precision and execution time. This paper's findings pave the way for generalized and transferable robotic systems manipulating various amorphous materials and automating a larger set of construction tasks and beyond.

ICRA Conference 2024 Conference Paper

Robot Trajectron: Trajectory Prediction-based Shared Control for Robot Manipulation

Pinhao Song
Pengteng Li
Erwin Aertbeliën
Renaud Detry

We address the problem of (a) predicting the trajectory of an arm reaching motion, based on a few seconds of the motion’s onset, and (b) leveraging this predictor to facilitate shared-control manipulation tasks, by reducing the operator’s cognitive load through assistance in their anticipated direction of motion. Our novel intent estimator, dubbed the Robot Trajectron (RT), produces a probabilistic representation of the robot’s anticipated trajectory based on its recent position, velocity and acceleration history. By taking arm dynamics into account, RT can capture the operator’s intent better than other SOTA models that only use the arm’s position, making it particularly well-suited to assist in tasks where the operator’s intent is susceptible to change. We derive a novel shared-control solution that combines RT’s predictive capacity to a representation of the locations of potential reaching targets. Our experiments demonstrate RT’s effectiveness in both intent estimation and shared-control tasks. We will make the code and data supporting our experiments publicly available at https://gitlab.kuleuven.be/detry-lab/public/robot-trajectron

IROS Conference 2017 Conference Paper

Task-oriented grasping with semantic and geometric scene understanding

Renaud Detry
Jeremie Papon
Larry H. Matthies

We present a task-oriented grasp model, that encodes grasps that are configurationally compatible with a given task. For instance, if the task is to pour liquid from a container, the model encodes grasps that leave the opening of the container unobstructed. The model consists of two independent agents: First, a geometric grasp model that computes, from a depth image, a distribution of 6D grasp poses for which the shape of the gripper matches the shape of the underlying surface. The model relies on a dictionary of geometric object parts annotated with workable gripper poses and preshape parameters. It is learned from experience via kinesthetic teaching. The second agent is a CNN-based semantic model that identifies grasp-suitable regions in a depth image: regions where a grasp will not impede the execution of the task. The semantic model allows us to encode relationships such as “grasp from the handle. ” A key element of this work is to use a deep network to integrate contextual task cues, and defer the structured-output problem of gripper pose computation to an explicit (learned) geometric model. Jointly, these two models generate grasps that are mechanically fit, and that grip on the object in a way that enables the intended task.

ICRA Conference 2016 Conference Paper

Probabilistic consolidation of grasp experience

Yasemin Bekiroglu
Andreas C. Damianou
Renaud Detry
Johannes A. Stork
Danica Kragic
Carl Henrik Ek

We present a probabilistic model for joint representation of several sensory modalities and action parameters in a robotic grasping scenario. Our non-linear probabilistic latent variable model encodes relationships between grasp-related parameters, learns the importance of features, and expresses confidence in estimates. The model learns associations between stable and unstable grasps that it experiences during an exploration phase. We demonstrate the applicability of the model for estimating grasp stability, correcting grasps, identifying objects based on tactile imprints and predicting tactile imprints from object-relative gripper poses. We performed experiments on a real platform with both known and novel objects, i. e. , objects the robot trained with, and previously unseen objects. Grasp correction had a 75% success rate on known objects, and 73% on new objects. We compared our model to a traditional regression model that succeeded in correcting grasps in only 38% of cases.

ICRA Conference 2015 Conference Paper

Learning the tactile signatures of prototypical object parts for robust part-based grasping of novel objects

Emil Hyttinen
Danica Kragic
Renaud Detry

We present a robotic agent that learns to derive object grasp stability from touch. The main contribution of our work is the use of a characterization of the shape of the part of the object that is enclosed by the gripper to condition the tactile-based stability model. As a result, the agent is able to express that a specific tactile signature may for instance indicate stability when grasping a cylinder, while cuing instability when grasping a box. We proceed by (1) discretizing the space of graspable object parts into a small set of prototypical shapes, via a data-driven clustering process, and (2) learning a touch-based stability classifier for each prototype. Classification is conducted through kernel logistic regression, applied to a low-dimensional approximation of the tactile data read from the robot's hand. We present an experiment that demonstrates the applicability of the method, yielding a success rate of 89%. Our experiment also shows that the distribution of tactile data differs substantially between grasps collected with different prototypes, supporting the use of shape cues in touch-based stability estimators.

ICRA Conference 2014 Conference Paper

Learning dexterous grasps that generalise to novel objects by combining hand and contact models

Marek Sewer Kopicki
Renaud Detry
Florian Schmidt 0001
Christoph Borst 0001
Rustam Stolkin
Jeremy L. Wyatt

Generalising dexterous grasps to novel objects is an open problem. We show how to learn grasps for high DoF hands that generalise to novel objects, given as little as one demonstrated grasp. During grasp learning two types of probability density are learned that model the demonstrated grasp. The first density type (the contact model) models the relationship of an individual finger part to local surface features at its contact point. The second density type (the hand configuration model) models the whole hand configuration during the approach to grasp. When presented with a new object, many candidate grasps are generated, and a kinematically feasible grasp is selected that maximises the product of these densities. We demonstrate 31 successful grasps on novel objects (an 86% success rate), transferred from 16 training grasps. The method enables: transfer of dexterous grasps within object categories; across object categories; to and from objects where there is no complete model of the object available; and using two different dexterous hands.

ICRA Conference 2014 Conference Paper

Representations for cross-task, cross-object grasp transfer

Martin Hjelm
Renaud Detry
Carl Henrik Ek
Danica Kragic

We address the problem of transferring grasp knowledge across objects and tasks. This means dealing with two important issues: 1) the induction of possible transfers, i. e. , whether a given object affords a given task, and 2) the planning of a grasp that will allow the robot to fulfill the task. The induction of object affordances is approached by abstracting the sensory input of an object as a set of attributes that the agent can reason about through similarity and proximity. For grasp execution, we combine a part-based grasp planner with a model of task constraints. The task constraint model indicates areas of the object that the robot can grasp to execute the task. Within these areas, the part-based planner finds a hand placement that is compatible with the object shape. The key contribution is the ability to transfer task parameters across objects while the part-based grasp planner allows for transferring grasp information across tasks. As a result, the robot is able to synthesize plans for previously unobserved task/object combinations. We illustrate our approach with experiments conducted on a real robot.

ICRA Conference 2013 Conference Paper

Learning a dictionary of prototypical grasp-predicting parts from grasping experience

Renaud Detry
Carl Henrik Ek
Marianna Madry
Danica Kragic

We present a real-world robotic agent that is capable of transferring grasping strategies across objects that share similar parts. The agent transfers grasps across objects by identifying, from examples provided by a teacher, parts by which objects are often grasped in a similar fashion. It then uses these parts to identify grasping points onto novel objects. We focus our report on the definition of a similarity measure that reflects whether the shapes of two parts resemble each other, and whether their associated grasps are applied near one another. We present an experiment in which our agent extracts five prototypical parts from thirty-two real-world grasp examples, and we demonstrate the applicability of the prototypical parts for grasping novel objects.

ICRA Conference 2013 Conference Paper

Sparse summarization of robotic grasping data

Martin Hjelm
Carl Henrik Ek
Renaud Detry
Hedvig Kjellström
Danica Kragic

We propose a new approach for learning a summarized representation of high dimensional continuous data. Our technique consists of a Bayesian non-parametric model capable of encoding high-dimensional data from complex distributions using a sparse summarization. Specifically, the method marries techniques from probabilistic dimensionality reduction and clustering. We apply the model to learn efficient representations of grasping data for two robotic scenarios.

IROS Conference 2013 Conference Paper

Unsupervised learning of predictive parts for cross-object grasp transfer

Renaud Detry
Justus H. Piater

We present a principled solution to the problem of transferring grasps across objects. Our approach identifies, through autonomous exploration, the size and shape of object parts that consistently predict the applicability of a grasp across multiple objects. The robot can then use these parts to plan grasps onto novel objects. By contrast to most recent methods, we aim to solve the part-learning problem without the help of a human teacher. The robot collects training data autonomously by exploring different grasps on its own. The core principle of our approach is an intensive encoding of low-level sensorimotor uncertainty with probabilistic models, which allows the robot to generalize the noisy autonomously-generated grasps. Object shape, which is our main cue for predicting grasps, is encoded with surface densities, that model the spatial distribution of points that belong to an object's surface. Grasp parameters are modeled with grasp densities, that correspond to the spatial distribution of object-relative gripper poses that lead to a grasp. The size and shape of grasp-predicting parts are identified by sampling the cross-object correlation of local shape and grasp parameters. We approximate sampling and integrals via Monte Carlo methods to make our computer implementation tractable. We demonstrate the applicability of our method in simulation. A proof of concept on a real robot is also provided.

ICRA Conference 2012 Conference Paper

Generalizing grasps across partly similar objects

Renaud Detry
Carl Henrik Ek
Marianna Madry
Justus H. Piater
Danica Kragic

The paper starts by reviewing the challenges associated to grasp planning, and previous work on robot grasping. Our review emphasizes the importance of agents that generalize grasping strategies across objects, and that are able to transfer these strategies to novel objects. In the rest of the paper, we then devise a novel approach to the grasp transfer problem, where generalization is achieved by learning, from a set of grasp examples, a dictionary of object parts by which objects are often grasped. We detail the application of dimensionality reduction and unsupervised clustering algorithms to the end of identifying the size and shape of parts that often predict the application of a grasp. The learned dictionary allows our agent to grasp novel objects which share a part with previously seen objects, by matching the learned parts to the current view of the new object, and selecting the grasp associated to the best-fitting part. We present and discuss a proof-of-concept experiment in which a dictionary is learned from a set of synthetic grasp examples. While prior work in this area focused primarily on shape analysis (parts identified, e. g. , through visual clustering, or salient structure analysis), the key aspect of this work is the emergence of parts from both object shape and grasp examples. As a result, parts intrinsically encode the intention of executing a grasp.

IROS Conference 2012 Conference Paper

Improving generalization for 3D object categorization with Global Structure Histograms

Marianna Madry
Carl Henrik Ek
Renaud Detry
Kaiyu Hang
Danica Kragic

We propose a new object descriptor for three dimensional data named the Global Structure Histogram (GSH). The GSH encodes the structure of a local feature response on a coarse global scale, providing a beneficial trade-off between generalization and discrimination. Encoding the structural characteristics of an object allows us to retain low local variations while keeping the benefit of global representativeness. In an extensive experimental evaluation, we applied the framework to category-based object classification in realistic scenarios. We show results obtained by combining the GSH with several different local shape representations, and we demonstrate significant improvements to other state-of-the-art global descriptors.

IROS Conference 2011 Conference Paper

Learning tactile characterizations of object- and pose-specific grasps

Yasemin Bekiroglu
Renaud Detry
Danica Kragic

Our aim is to predict the stability of a grasp from the perceptions available to a robot before attempting to lift up and transport an object. The percepts we consider consist of the tactile imprints and the object-gripper configuration read before and until the robot's manipulator is fully closed around an object. Our robot is equipped with multiple tactile sensing arrays and it is able to track the pose of an object during the application of a grasp. We present a kernel-logistic-regression model of pose- and touch-conditional grasp success probability which we train on grasp data collected by letting the robot experience the effect on tactile and visual signals of grasps suggested by a teacher, and letting the robot verify which grasps can be used to rigidly control the object. We consider models defined on several subspaces of our input data - e. g. , using tactile perceptions or pose information only. Our experiment demonstrates that joint tactile and pose-based perceptions carry valuable grasp-related information, as models trained on both hand poses and tactile parameters perform better than the models trained exclusively on one perceptual input.

IROS Conference 2010 Conference Paper

Learning probabilistic discriminative models of grasp affordances under limited supervision

Ayse Erkan
Oliver Kroemer
Renaud Detry
Yasemin Altun
Justus H. Piater
Jan Peters 0001

This paper addresses the problem of learning and efficiently representing discriminative probabilistic models of object-specific grasp affordances particularly when the number of labeled grasps is extremely limited. The proposed method does not require an explicit 3D model but rather learns an implicit manifold on which it defines a probability distribution over grasp affordances. We obtain hypothetical grasp configurations from visual descriptors that are associated with the contours of an object. While these hypothetical configurations are abundant, labeled configurations are very scarce as these are acquired via time-costly experiments carried out by the robot. Kernel logistic regression (KLR) via joint kernel maps is trained to map the hypothesis space of grasps into continuous class-conditional probability values indicating their achievability. We propose a soft-supervised extension of KLR and a framework to combine the merits of semi-supervised and active learning approaches to tackle the scarcity of labeled grasps. Experimental evaluation shows that combining active and semi-supervised learning is favorable in the existence of an oracle. Furthermore, semi-supervised learning outperforms supervised learning, particularly when the labeled data is very limited.

ICRA Conference 2010 Conference Paper

Refining grasp affordance models by experience

Renaud Detry
Dirk Kraft
Anders Glent Buch
Norbert Krüger
Justus H. Piater

We present a method for learning object grasp affordance models in 3D from experience, and demonstrate its applicability through extensive testing and evaluation on a realistic and largely autonomous platform. Grasp affordance refers here to relative object-gripper configurations that yield stable grasps. These affordances are represented probabilistically with grasp densities, which correspond to continuous density functions defined on the space of 6D gripper poses. A grasp density characterizes an object's grasp affordance; densities are linked to visual stimuli through registration with a visual model of the object they characterize. We explore a batch-oriented, experience-based learning paradigm where grasps sampled randomly from a density are performed, and an importance-sampling algorithm learns a refined density from the outcomes of these experiences. The first such learning cycle is bootstrapped with a grasp density formed from visual cues. We show that the robot effectively applies its experience by downweighting poor grasp solutions, which results in increased success rates at subsequent learning cycles. We also present success rates in a practical scenario where a robot needs to repeatedly grasp an object lying in an arbitrary pose, where each pose imposes a specific reaching constraint, and thus forces the robot to make use of the entire grasp density to select the most promising achievable grasp.

IROS Conference 2009 Conference Paper

Active learning using mean shift optimization for robot grasping

Oliver Kroemer
Renaud Detry
Justus H. Piater
Jan Peters 0001

When children learn to grasp a new object, they often know several possible grasping points from observing a parent's demonstration and subsequently learn better grasps by trial and error. From a machine learning point of view, this process is an active learning approach. In this paper, we present a new robot learning framework for reproducing this ability in robot grasping. For doing so, we chose a straightforward approach: first, the robot observes a few good grasps by demonstration and learns a value function for these grasps using Gaussian process regression. Subsequently, it chooses grasps which are optimal with respect to this value function using a mean-shift optimization approach, and tries them out on the real system. Upon every completed trial, the value function is updated, and in the following trials it is more likely to choose even better grasping points. This method exhibits fast learning due to the data-efficiency of the Gaussian process regression framework and the fact that the mean-shift method provides maxima of this cost function. Experiments were repeatedly carried out successfully on a real robot system. After less than sixty trials, our system has adapted its grasping policy to consistently exhibit successful grasps.