Dana Kulic

IROS Conference 2025 Conference Paper

Evaluating Human-Robot Collaboration through Online Video: Perspective Matters

• Leimin Tian
• Shiyu Xu
• Kerry He
• Rachel Love
• Akansel Cosgun
• Dana Kulic

Online evaluation is increasingly adopted in robotics research, providing an efficient approach to collect data from large and diverse populations. However, there have been ongoing debates about online studies as a proxy for in-person studies, especially where a participant passively observes video of robot behaviours or interaction. We conduct an online video comparison study (N=178) evaluating three robot handover policies in a collaborative assembly task, namely an adaptive autonomous policy, a non-adaptive scripted policy, and teleoperation. Participants watched three sets of videos in third-person view, each consisting of 9 sequential handovers executing one of the policies. Compared to in-person participants in two previous studies who evaluated handovers as users, online participants were observant of different robot behaviours and human-robot collaboration contexts, with 76. 4% and 71. 9% recognising the adaptive handovers exhibited by the teleoperated and autonomous robot, respectively. However, as observers, online participants showed more critical subjective perceptions compared to the in-person participants with a user’s perspective. They valued efficiency over adaptation with twice more autonomous handovers rated as being too late compared to scripted handovers. Our work highlights the need to consider user contexts when evaluating human-robot collaboration.

AAMAS Conference 2025 Conference Paper

Explaining Facial Expression Recognition

• Sanjeev Nahulanthran
• Leimin Tian
• Dana Kulic
• Mor Vered

Facial expression recognition (FER) has emerged as a promising approach to the development of emotion-aware intelligent agents and systems. However, key challenges remain in utilizing FER in real-world contexts, including ensuring user understanding and establishing a suitable level of user trust. We developed a novel explanation method utilizing Facial Action Units (FAUs) to explain the output of a FER model through both textual and visual modalities. We conducted an empirical user study evaluating user understanding and trust, comparing our approach to stateof-the-art eXplainable AI (XAI) methods. Our results indicate that visual AND textual as well as textual-only FAU-based explanations resulted in better user understanding of the FER model. We also show that all modalities of FAU-based methods improved appropriate trust of the users towards the FER model.

IROS Conference 2024 Conference Paper

A comparison of audible, visual, and multi-modal communication for multi-robot supervision and situational awareness

• Richard Attfield
• Elizabeth A. Croft
• Dana Kulic

Multi-robot supervision becomes increasingly cognitively demanding as the ratio of robots to human supervisors rises, potentially leading to situational awareness (SA) losses and robot system failures. Nonverbal cues have been employed to direct supervisor attention and prevent awareness loss in diverse human-computer interaction (HCI) settings. This paper compares the effects of uni-modal and multi-modal audiovisual nonverbal cues on supervisor SA in a multi-robot supervision task. In a simulation-based navigation scenario, 50 participants monitored a multi-robot mission and responded to supervision requests from the robots. We evaluated supervisor SA using response speed and the situational awareness global assessment technique. Results demonstrate that supervisor awareness hinges on the communication method employed by the robots, with greater significance observed at higher awareness levels and when the robot-to-human ratio is higher. Findings also indicate the utility of sonification mapping in human-multirobot interactions and the benefits of multi-modal cues for sustaining awareness during multi-robot supervision.

IROS Conference 2024 Conference Paper

Occlusion Handling by Pushing for Enhanced Fruit Detection

• Ege Gursoy
• Dana Kulic
• Andrea Cherubini

In agricultural robotics, effective observation and localization of fruits present challenges due to occlusions caused by other parts of the tree, such as branches and leaves. These occlusions can result in false fruit localization or impede the robot from picking the fruit. The objective of this work is to push away branches that block the fruit’s view to increase their visibility. Our setup consists of an RGB-D camera and a robot arm. First, we detect the occluded fruit in the RGB image and estimate its occluded part via a deep learning generative model in the depth space. The direction to push to clear the occlusions is determined using classic image processing techniques. We then introduce a 3D extension of the 2D Hough transform to detect straight line segments in the point cloud. This extension helps detect tree branches and identify the one mainly responsible for the occlusion. Finally, we clear the occlusion by pushing the branch with the robot arm. Our method uses a combination of deep learning for fruit appearance estimation, classic image processing for push direction determination, and 3D Hough transform for branch detection. We validate our perception methods through real data under different lighting conditions and various types of fruits (i. e. apple, lemon, orange), achieving improved visibility and successful occlusion clearance. We demonstrate the practical application of our approach through a real robot branch pushing demonstration.

PRL Workshop 2024 Workshop Paper

POSGGym: A Library for Decision-Theoretic Planning and Learning in Partially Observable, Multi-Agent Environments

• Jonathon Schwartz
• Rhys Newbury
• Dana Kulic
• Hanna Kurniawati

Seamless integration of Planning Under Uncertainty and Reinforcement Learning (RL) promises to bring the best of both model-driven and data-driven worlds to multi-agent decisionmaking, resulting in an approach with assurances on performance that scales well to more complex problems. Despite this potential, progress in developing such methods has been hindered by the lack of adequate evaluation and simulation platforms. Researchers have had to rely on creating custom environments, which reduces efficiency and makes comparing new methods difficult. In this paper, we introduce POSGGym: a library for facilitating planning and RL research in partially observable, multi-agent domains. It provides a diverse collection of discrete and continuous environments, complete with their dynamics models and a reference set of policies that can be used to evaluate generalization to novel partners. Leveraging POSGGym, we empirically investigate existing state-of-the-art planning methods and a method that combines planning and RL in the type-based reasoning setting. Our experiments corroborate that combining planning and RL can yield superior performance compared to planning or RL alone, given the model of the environment and other agents is correct. However, our particular setup also reveals that this integrated approach could result in worse performance when the model of other agents is incorrect. Our findings indicate the benefit of integrating planning and RL in partially observable, multi-agent domains, while serving to highlight several important directions for future research. Code available at: https: //github. com/RDLLab/posggym.

IROS Conference 2023 Conference Paper

Rotating Objects via in-Hand Pivoting Using Vision, Force and Touch

• Shiyu Xu
• Tianyuan Liu
• Michael Wong
• Dana Kulic
• Akansel Cosgun

We propose a robotic manipulation method that can pivot objects on a surface using vision, wrist force and tactile sensing. We aim to control the rotation of an object around the grip point of a parallel gripper by allowing rotational slip, while maintaining a desired wrist force profile. Our approach runs an end-effector position controller and a gripper width controller concurrently in a closed loop. The position controller maintains a desired force using vision and wrist force. The gripper controller uses tactile sensing to keep the grip firm enough to prevent translational slip, but loose enough to allow rotational slip. Our sensor-based control approach relies on matching a desired force profile derived from object dimensions and weight, as well as vision-based monitoring of the object pose. The gripper controller uses tactile sensors to detect and prevent translational slip by tightening the grip when needed. Experimental results where the robot was tasked with rotating cuboid objects 90 degrees show that the multi-modal pivoting approach was able to rotate the objects without causing lift or translational slip, and was more energy-efficient compared to using a single sensor modality or pick-and-place.

IS Journal 2022 Journal Article

Fast Approximate Multioutput Gaussian Processes

• Vladimir Joukov
• Dana Kulic

Gaussian processes regression models are an appealing machine learning method as they learn expressive nonlinear models from exemplar data with minimal parameter tuning and estimate both the mean and covariance of unseen points. However, cubic computational complexity growth with the number of samples has been a long standing challenge. Training requires the inversion of $N \times N$N×N kernel at every iteration, whereas regression needs computation of an $m \times N$m×N kernel, where $N$N and $m$m are the number of training and test points, respectively. This work demonstrates how approximating the covariance kernel using eigenvalues and functions leads to an approximate Gaussian process with significant reduction in training and regression complexity. Training now requires computing only an $N \times n$N×n eigenfunction matrix and an $n \times n$n×n inverse, where $n$n is a selected number of eigenvalues. Furthermore, regression now only requires an $m \times n$m×n matrix. Finally, in a special case, the hyperparameter optimization is completely independent from the number of training samples. The proposed method can regress over multiple outputs, learn the correlations between them, and estimate their derivatives to any order. The computational complexity reduction, regression capabilities, multioutput correlation learning, and comparison to the state of the art are demonstrated in simulation examples. Finally we show how the proposed approach can be utilized to model real human data.

IROS Conference 2021 Conference Paper

Human Motion Imitation using Optimal Control with Time-Varying Weights

• Shouyo Ishida
• Tatsuki Harada
• Pamela Carreno-Medrano
• Dana Kulic
• Gentiane Venture

Research in biomechanics hypothesizes that human motion is optimal with respect to an unknown cost function that varies depending on the action and/or task. This unknown cost function is often approximated as the weighted sum of a set of features or basis cost functions. As a person performs a sequence of actions, the weights associated to each of these basis functions are likely to vary over time. Given a human demonstration and the corresponding cost weight trajectory recovered via inverse optimal control (IOC), this paper proposes an optimal control (OC) method that can generate robot motion based on human movement using time-varying cost function weights. By using time-varying weights, the proposed optimal control method can handle changing optimization criteria without segmentation. The method is evaluated both in simulation and with recorded human data. Using human demonstration data, we demonstrate the reproduction of pick-and-place motions with an average end-effector error at the pick place location within 0. 82 cm, which is significantly lower than the average trajectory error, indicating that the approach correctly prioritizes reaching the pick and place locations without manual segmentation.

IROS Conference 2021 Conference Paper

Memory-based Deep Reinforcement Learning for POMDPs

• Lingheng Meng
• Rob Gorbet
• Dana Kulic

A promising characteristic of Deep Reinforcement Learning (DRL) is its capability to learn optimal policy in an end-to-end manner without relying on feature engineering. However, most approaches assume a fully observable state space, i. e. fully observable Markov Decision Processes (MDPs). In real-world robotics, this assumption is unpractical, because of issues such as sensor sensitivity limitations and sensor noise, and the lack of knowledge about whether the observation design is complete or not. These scenarios lead to Partially Observable MDPs (POMDPs). In this paper, we propose Long-Short-Term-Memory-based Twin Delayed Deep Deterministic Policy Gradient (LSTM-TD3) by introducing a memory component to TD3, and compare its performance with other DRL algorithms in both MDPs and POMDPs. Our results demonstrate the significant advantages of the memory component in addressing POMDPs, including the ability to handle missing and noisy observation data.

IROS Conference 2020 Conference Paper

A Framework for Human-Robot Interaction User Studies

• Vidyasagar Rajendran
• Pamela Carreno-Medrano
• Wesley Fisher
• Alexander Werner
• Dana Kulic

Human-Robot Interaction (HRI) user studies are challenging to evaluate and compare due to a lack of standardization and the infrastructure required to implement each study. The lack of experimental infrastructure also makes it difficult to systematically evaluate the impact of individual components (e. g. , the quality of perception software) on overall system performance. This work proposes a framework to ease the implementation and reproducibility of human-robot interaction user studies. The framework utilizes ROS middleware and is implemented with four modules: perception, decision, action, and metrics. The perception module aggregates sensor data to be used by the decision and action modules. The decision module is the task-level executive and can be designed by the HRI researcher for their specific task. The action module takes subtask requests from the decision module and breaks them down into motion primitives for execution on the robot. The metrics module tracks and generates quantitative metrics for the study. The framework is implemented with modular interfaces to allow for alternate implementations within each module and can be generalized for a variety of tasks and human/robot roles. The framework is illustrated through an example scenario involving a human and a Franka Emika Panda arm collaboratively assembling a toolbox together.

IROS Conference 2020 Conference Paper

Active Preference Learning using Maximum Regret

• Nils Wilde
• Dana Kulic
• Stephen L. Smith 0001

We study active preference learning as a frame-work for intuitively specifying the behaviour of autonomous robots. A user chooses the preferred behaviour from a set of alternatives, from which the robot learns the user's preferences, modeled as a parameterized cost function. Previous approaches present users with alternatives that minimize the uncertainty over the parameters of the cost function. However, different parameters might lead to the same optimal behaviour; as a consequence the solution space is more structured than the parameter space. We exploit this by proposing a query selection that greedily reduces the maximum error ratio over the solution space. In simulations we demonstrate that the proposed approach outperforms other state of the art techniques in both learning efficiency and ease of queries for the user. Finally, we show that evaluating the learning based on the similarities of solutions instead of the similarities of weights allows for better predictions for different scenarios.

ICRA Conference 2020 Conference Paper

Learning User Preferences from Corrections on State Lattices

• Nils Wilde
• Dana Kulic
• Stephen L. Smith 0001

Enabling a broader range of users to efficiently deploy autonomous mobile robots requires intuitive frameworks for specifying a robot's task and behaviour. We present a novel approach using learning from corrections (LfC), where a user is iteratively presented with a solution to a motion planning problem. Users might have preferences about parts of a robot's environment that are suitable for robot traffic or that should be avoided as well as preferences on the control actions a robot can take. The robot is initially unaware of these preferences; thus, we ask the user to provide a correction to the presented path. We assume that the user evaluates paths based on environment and motion features. From a sequence of corrections we learn weights for these features, which are then considered by the motion planner, resulting in future paths that better fit the user's preferences. We prove completeness of our algorithm and demonstrate its performance in simulations. Thereby, we show that the learned preferences yield good results not only for a set of training tasks but also for test tasks, as well as for different types of user behaviour.

IROS Conference 2020 Conference Paper

Supportive Actions for Manipulation in Human-Robot Coworker Teams

• Shray Bansal
• Rhys Newbury
• Wesley P. Chan
• Akansel Cosgun
• Aimee Allen
• Dana Kulic
• Tom Drummond
• Charles Isbell

The increasing presence of robots alongside humans, such as in human-robot teams in manufacturing, gives rise to research questions about the kind of behaviors people prefer in their robot counterparts. We term actions that support interaction by reducing future interference with others as supportive robot actions and investigate their utility in a co-located manipulation scenario. We compare two robot modes in a shared table pick-and-place task: (1) Task-oriented: the robot only takes actions to further its task objective and (2) Supportive: the robot sometimes prefers supportive actions to task-oriented ones when they reduce future goal-conflicts. Our experiments in simulation, using a simplified human model, reveal that supportive actions reduce the interference between agents, especially in more difficult tasks, but also cause the robot to take longer to complete the task. We implemented these modes on a physical robot in a user study where a human and a robot perform object placement on a shared table. Our results show that a supportive robot was perceived more favorably as a coworker and also reduced interference with the human in one of two scenarios. However, it also took longer to complete the task highlighting an interesting trade-off between task-efficiency and human-preference that needs to be considered before designing robot behavior for close-proximity manipulation scenarios.

ICRA Conference 2018 Conference Paper

Learning User Preferences in Robot Motion Planning Through Interaction

• Nils Wilde
• Dana Kulic
• Stephen L. Smith 0001

In this paper we develop an approach for learning user preferences for complex task specifications through human-robot interaction. We consider the problem of planning robot motion in a known environment, but where a user has specified additional spatial and temporal constraints on allowable robot motions. To illustrate the impact of the user's constraints on performance, we iteratively present users with alternative solutions on an interface. The user provides a ranking of alternate paths, and from this we learn about the importance of different constraints. This allows for an accessible method for specifying complex robot tasks. We present an algorithm that iteratively builds a set of constraints on the relative importance of each user constraint, and prove that with sufficient interaction, the algorithm determines a user-optimal path. We demonstrate the practical performance by simulating realistic material transport scenarios in industrial facilities.

ICRA Conference 2018 Conference Paper

Spherical Foot Placement Estimator for Humanoid Balance Control and Recovery

• Brandon J. DeHart
• Rob Gorbet
• Dana Kulic

One of the main challenges of bipedal gait is to avoid falling due to unknown disturbances. Compensating for these disturbances in bipeds is often achieved by leaning or stepping. In this work, the Spherical Foot Placement Estimator (SFPE) is introduced, which uses the biped's current kinematics and dynamics to predict if a step is needed, and if so where to step, to restore balance in 3D. An example of a controller using the SFPE is shown, which augments an existing optimal controller with both leaning and stepping: SFPE-based feedback is used to generate a desired momentum for momentum-based leaning while the SFPE point is used as a control reference for stepping. The new estimator outperforms existing balance criteria by providing both recovery step location prediction and momentum objectives with smooth dynamics.

IROS Conference 2017 Conference Paper

Exercise motion classification from large-scale wearable sensor data using convolutional neural networks

• Terry Taewoong Um
• Vahid Babakeshizadeh
• Dana Kulic

The ability to accurately identify human activities is essential for developing automatic rehabilitation and sports training systems. In this paper, large-scale exercise motion data obtained from a forearm-worn wearable sensor are classified with a convolutional neural network (CNN). Time-series data consisting of accelerometer and orientation measurements are formatted as images, allowing the CNN to automatically extract discriminative features. A comparative study on the effects of image formatting and different CNN architectures is also presented. The best performing configuration classifies 50 gym exercises with 92. 1% accuracy.

IROS Conference 2017 Conference Paper

Generalized Hebbian algorithm for wearable sensor rotation estimation

• Vladimir Joukov
• Jonathan Feng-Shun Lin
• Dana Kulic

Inertial measurement units (IMUs) enable human motion measurement in any environment, which can be useful for human robot interaction, exoskeletons, and active prosthetics. This paper proposes an approach for estimating the orientation between a wearable IMU sensor and the body frame of the wearer using a simple and fast calibration procedure. The proposed approach uses the generalized Hebbian algorithm to incrementally estimate the axis aligned with gravity using acceleration measurements obtained during a static pose, and the axis perpendicular to the saggital plane using gyro measurements obtained during sagittal plane movements. An automated convergence criterion based on the sensor measurement variance is used. The proposed approach is tested in simulation and with human movement and demonstrates excellent and fast calibration performance.

IROS Conference 2017 Conference Paper

Human motion estimation on Lie groups using IMU measurements

• Vladimir Joukov
• Josip Cesic
• Kevin Westermann
• Ivan Markovic
• Dana Kulic
• Ivan Petrovic

This paper proposes a new algorithm for human motion estimation using inertial measurement unit (IMU) measurements. We model the joints by matrix Lie groups, namely the special orthogonal groups SO(2) and SO(3), representing rotations in 2D and 3D space, respectively. The state space is defined by the Cartesian product of the rotation groups and their velocities and accelerations, given a kinematic model of the articulated body. In order to estimate the state, we propose the Lie Group Extended Kalman Filter (LG-EKF), thus explicitly accounting for the non-Euclidean geometry of the state space, and we derive the LG-EKF recursion for articulated motion estimation based on IMU measurements. The performance of the proposed algorithm is compared to the EKF based on Euler angle parametrization in both simulation and real-world experiments. The results show that for motion near gimbal lock regions, which is common for shoulder movement, the proposed filter is a significant improvement over the Euler angles EKF.

AAMAS Conference 2016 Conference Paper

Expectation-Maximization for Inverse Reinforcement Learning with Hidden Data

• Kenneth Bogert
• Jonathan Feng-Shun Lin
• Prashant Doshi
• Dana Kulic

IROS Conference 2015 Conference Paper

Constrained dynamic parameter estimation using the Extended Kalman Filter

• Vladimir Joukov
• Vincent Bonnet
• Gentiane Venture
• Dana Kulic

In this paper we present a real-time method for identification of the dynamic parameters of a manipulator and its load using kinematic measurements and either joint torques or force and moment at the base. The parameters are estimated using the Extended Kalman Filter and constraints are imposed using Sigmoid functions to ensure the parameters remain within their physically feasible ranges, such as links having positive masses and moments of inertia. Identified parameters can be used in model based controllers. The presented approach is validated through simulation and on data collected with the Barret WAM manipulator. Using the estimated parameters instead of ones provided by the manufacturer greatly improves joint torque prediction.

IROS Conference 2015 Conference Paper

Control of soft pneumatic finger-like actuators for affective motion generation

• Mohammadreza Memarian
• Rob Gorbet
• Dana Kulic

This paper investigates the design and implementation of a finger-like robotic structure capable of reproducing human hand gestural movements performed by a multi-fingered, hand-like structure. In this work, we present a pneumatic circuit and a closed-loop controller for a finger-like soft pneumatic actuator. Experimental results demonstrate the performance of the pneumatic and control systems of the soft pneumatic actuator, and its ability to track human movement trajectories with affective content.

IROS Conference 2015 Conference Paper

Curiosity-Based Learning Algorithm for distributed interactive sculptural systems

• Matthew T. K. Chan
• Rob Gorbet
• Philip Beesley
• Dana Kulic

The ability to engage human observers is a key requirement for both social robots and the arts. In this paper, we propose an approach for adapting the Intelligent Adaptive Curiosity learning algorithm to distributed interactive sculptural systems. This Curiosity-Based Learning Algorithm (CBLA) allows the system to learn about its own mechanisms and its surroundings through self-experimentation and interaction. A novel formulation using multiple agents as learning subsets of the system that communicate through shared input variables enables us to scale to a much larger system with diverse types of sensors and actuators. Experiments on a prototype interactive sculpture demonstrate the exploratory patterns of the CBLA and collective learning behaviours through the integration of multiple learning agents.

IROS Conference 2015 Conference Paper

Modelling and experimental analysis of a novel design for soft pneumatic artificial muscles

• Mohammadreza Memarian
• Rob Gorbet
• Dana Kulic

Soft pneumatic artificial muscles (SPAMs) are a type of pneumatic actuator that provide customizable motion trajectories in three dimensional space without the need for rigid links or a transmission mechanism. This paper presents a novel design for producing SPAMs, named wrapped SPAMs (WSPAMs). Unlike previous SPAM designs, the production process of WSPAM is highly repeatable, while the motion trajectory can be easily modified. A model for predicting the steady-state angular displacement of a WSPAM actuator based on its geometrical parameters and the elasticity of the materials used in its production is presented and experimentally validated.

ICRA Conference 2014 Conference Paper

Push recovery and online gait generation for 3D bipeds with the foot placement estimator

• Brandon J. DeHart
• Dana Kulic

Humanoid robots have many potential applications in man-made environments, including performing hazardous tasks, assisting the elderly, and as a replacement for our aging workforce. However, generating a reliable gait for biped robots is challenging, particularly for dynamic gait and in the presence of unknown external disturbances, such as a bump from someone walking by. In this work, a 3D formulation of the Foot Placement Estimator is used with a high-level control strategy to achieve a dynamic gait capable of handling external disturbances. A key benefit of this approach is that the robot is able to respond in real time to external disturbances regardless of whether it is at rest or in motion. This strategy is implemented in simulation to control a 14-DOF lower-body humanoid robot being subjected to unknown external forces, both when at rest and while walking, and shown to generate stabilizing stepping actions.

ICRA Conference 2013 Conference Paper

Gait generation via the Foot Placement Estimator for 3D bipedal robots

• Safwan Choudhury
• Dana Kulic

This paper proposes a trajectory generation and control strategy for generating stable gait subject to unknown disturbances, based on the concept of the Foot Placement Estimator (FPE). While most walking control strategies in the field of bipedal locomotion aim to constantly maintain balance, the Foot Placement Estimator (FPE) estimates where the foot must be placed in order to restore balance. One of the key novelties of the FPE approach is its natural extension to form complete gait cycles using a state machine and simple proportional-derivative controllers. In this paper, the FPE control strategy is extended from 2D to 3D robots, and demonstrated in simulation on a 14-DOF lower body bipedal robot.

IROS Conference 2013 Conference Paper

Robust path following for robot manipulators

• Rajan J. Gill
• Dana Kulic
• Christopher Nielsen

Path following controllers make the output of a control system approach and traverse a pre-specified path with no a priori time-parametrization. This paper implements a path following controller, based on transverse feedback linearization (TFL), which guarantees invariance of the path to be followed. The coordinate and feedback transformation employed allows one to easily design control laws to generate arbitrary desired motions on the path for the closed-loop system. The approach is applied to an uncertain and simplified model of a robot manipulator for which none of the dynamic parameters are measured. The controller is made robust to modelling uncertainties using Lyapunov redesign. The robustified controller is tested on a 4-degree-of-freedom (4-DOF) manipulator with a combination of revolute and linear actuated links. The experimental results show a substantial improvement when using the robust controller for path following versus standard state feedback.

IROS Conference 2012 Conference Paper

Online learning of inverse dynamics via Gaussian Process Regression

• Joseph Sun de la Cruz
• William S. Owen
• Dana Kulic

Model-based control strategies for robot manipulators can present numerous performance advantages when an accurate model of the system dynamics is available. In practice, obtaining such a model is a challenging task which involves modeling such physical processes as friction, which may not be well understood and difficult to model. This paper proposes an approach for online learning of the inverse dynamics model using Gaussian Process Regression. The Sparse Online Gaussian Process (SOGP) algorithm is modified to allow for incremental updates of the model and hyperparameters. The influence of initialization on the performance of the learning algorithms, based on any a-priori knowledge available, is also investigated. The proposed approach is compared to existing learning and fixed control algorithms and shown to be capable of fast initialization and learning rate.

IROS Conference 2010 Conference Paper

A stereo camera based full body human motion capture system using a partitioned particle filter

• Zhenning Li 0006
• Dana Kulic

In this paper, we propose a marker-less full body human motion capture system designed for humanoid robot applications. The system is based on a stereo camera, and therefore has strong portability. Tracking is implemented within the particle filter framework, and the high dimensionality problem is solved through partitioned sampling. Taking advantage of the stereo setup, we propose a depth cue which resolves the problem of missing depth information in monocular tracking. Three other cues, the edge cue, the color cue and the distance cue, are also integrated into the system to enhance the tracking performance. The system is tested using the publicly available CMU MOCAP database which also includes ground truth data, and this enables us to analyze the results quantitatively and compare the relative usefulness of different cues. The system is shown to be capable of tracking challenging videos accurately and robustly in near real-time.

IROS Conference 2010 Conference Paper

Incremental learning of human behaviors using hierarchical hidden Markov models

• Dana Kulic
• Yoshihiko Nakamura

This paper proposes a novel approach for extracting a model of movement primitives and their sequential relationships during online observation of human motion. In the proposed approach, movement primitives, modeled as hidden Markov models, are autonomously segmented and learned incrementally during observation. At the same time, a higher abstraction level hidden Markov model is also learned, encapsulating the relationship between the movement primitives. For the higher level model, each hidden state represents a motion primitive, and the observation function is based on the likelihood that the observed data is generated by the motion primitive model. An approach for incremental training of the higher order model during online observation is developed. The approach is validated on a dataset of continuous movement data.

IROS Conference 2010 Conference Paper

What do you expect from a robot that tells your future? The crystal ball

• Wataru Takano
• Hirotaka Imagawa
• Dana Kulic
• Yoshihiko Nakamura

This paper proposes an approach to hierarchy formation of human behaviors, extraction of the behavioral transitions, and their application to prediction and automatic generation of behaviors. Human demonstrator motion patterns are stored as motion symbols, which abstract the motion data by using Hidden Markov Models. The stored motion patterns are organized into a hierarchical tree structure, which represents the similarity among the motion patterns and provides abstracted motion patterns. Concatenated sequences of motion patterns are stochastically represented as transitions between the abstracted motion patterns by using an Ngram Model, and the transitional relationships of the human behaviors are extracted. The behavioral hierarchy and transition model make it possible to predict human behaviors during observation and to generate sequences of motion patterns automatically while maintaining a natural motion stream, as if the system is a “crystal ball” to reflect future behaviors. The experiments validates the proposed framework by using a developed visualization system, which shows the demonstrator or the operator the established hierarchical tree and the transition network of the motion patterns, predicted behaviors and generated sequences of the motion patterns.

IROS Conference 2009 Conference Paper

Comparative study of representations for segmentation of whole body human motion data

• Dana Kulic
• Yoshihiko Nakamura

In previous work, the authors have been developing a stochastic model based approach for on-line segmentation of whole body human motion patterns during human motion observation and learning, using a simplified kinematic model of the human body. In this paper, we extend the proposed approach to larger, more realistic kinematic models, which can better represent a larger variety of human motions. These larger models may include spherical in addition to revolute joints. We examine the effects on segmentation performance due to motion representation choice, and compare the segmentation efficacy when Cartesian or joint angle data is used. The approach is tested on whole body human motion data modeled with a 42DoF kinematic model. The results indicate that Cartesian data seems to correspond most closely to the human evaluation of segment points. The experiments also demonstrate the efficacy of the segmentation approach for large kinematic models and a variety of human motions.

IROS Conference 2009 Conference Paper

Evaluation of affective state estimations using an on-line reporting device during human-robot interactions

• Susana Zoghbi
• Elizabeth A. Croft
• Dana Kulic
• H. F. Machiel Van der Loos

In order to develop a friendly and safe interaction between humans and robots, it is essential for the robot to evaluate user's affective states and respond accordingly. However, affective states are typically assessed using offline questionnaires and user reports. In this paper we investigate the use of an online-device for collecting real-time user reports of affective state during interaction with a robot. These reports are compared to both previous survey reports taken after the interaction, and the affective states estimated by an inference system. The aim is to evaluate and characterize the physiological signal-based inference system and determine which factors significantly influence its performance. This analysis will be used in future work, to fine tune the affective estimations by identifying what kind of variations in physiological signals precede or accompany the variations in reported affective states.

ICRA Conference 2009 Conference Paper

Whole body motion primitive segmentation from monocular video

• Dana Kulic
• Dongheui Lee
• Yoshihiko Nakamura

This paper proposes a novel approach for motion primitive segmentation from continuous full body human motion captured on monocular video. The proposed approach does not require a kinematic model of the person, nor any markers on the body. Instead, optical flow computed directly in the image plane is used to estimate the location of segment points. The approach is based on detecting tracking features in the image based on the Shi and Thomasi algorithm [1]. The optical flow at each feature point is then estimated using the Lucas Kanade Pyramidal Optical Flow estimation algorithm [2]. The feature points are clustered and tracked on-line to find regions of the image with coherent movement. The appearance and disappearance of these coherent clusters indicates the start and end points of motion primitive segments. The algorithm performance is validated on full body motion video sequences, and compared to a joint-angle, motion capture based approach. The results show that the segmentation performance is comparable to the motion capture based approach, while using much simpler hardware and at a lower computational effort.

ICRA Conference 2008 Conference Paper

Combining automated on-line segmentation and incremental clustering for whole body motions

• Dana Kulic
• Wataru Takano
• Yoshihiko Nakamura

This paper describes a novel approach for incremental learning of human motion pattern primitives through on-line observation of human motion. The observed motion time series data stream is first stochastically segmented into potential motion primitive segments, based on the assumption that data belonging to the same motion primitive will have the same underlying distribution. The motion segments are then abstracted into a stochastic model representation, and automatically clustered and organized. As new motion patterns are observed, they are incrementally grouped together based on their relative distance in the model space. The resulting representation of the knowledge domain is a tree structure, with specialized motions at the tree leaves, and generalized motions closer to the root. The tree leaves, which represent the most specialized learned motion primitives, are then passed back to the segmentation algorithm, so that as the number of known motion primitives increases, the accuracy of the segmentation can also be improved. The combined algorithm is tested on a sequence of continuous human motion data obtained through motion capture, and demonstrates the performance of the proposed approach.

ICRA Conference 2008 Conference Paper

Missing motion data recovery using factorial hidden Markov models

• Dongheui Lee
• Dana Kulic
• Yoshihiko Nakamura

This paper proposes a method to recover missing data during observation by factorial hidden Markov models (FHMMs). The fundamental idea of the proposed method originates from the mimesis model, inspired by the mirror neuron system. By combining the motion recognition from partial observation algorithm and the proto-symbol based duplication of observed motion algorithm, whole body motion imitation from partial observation can be achieved. The algorithm for missing data recovery uses the same basic strategy as the whole body motion imitation from partial observation, but requires more accurate spatial representability. FHMMs allow for more efficient representation of a continuous data sequence by distributed state representation compared to hidden Markov models (HMMs). The proposed algorithm is tested with human motion data and the experimental results show improved representability compared to the conventional HMMs.

IROS Conference 2008 Conference Paper

Scaffolding on-line segmentation of full body human motion patterns

• Dana Kulic
• Yoshihiko Nakamura

This paper develops an approach for on-line segmentation of whole body human motion patterns during human motion observation and learning. A Hidden Markov Model is used to represent the incoming data sequence, where each model state represents the probability density estimate over a window of the data. Based on the assumption that data belonging to the same motion primitive will have the same underlying distribution, the segmentation is implemented by finding the optimum state sequence over the developed model. The basic algorithm is modified to add the capability for modifying the model based on known motion primitives. The inclusion of such scaffolding motion primitives can improve the performance of the basic segmentation algorithm. The modified algorithm is tested on a corpus of continuous human motion data to show the efficacy of the proposed approach.

IROS Conference 2007 Conference Paper

Dynamic parameter identification for the CRS A460 robot

• Katayon Radkhah
• Dana Kulic
• Elizabeth A. Croft

Dynamic Parameter Identification is a useful tool for developing and evaluating robot control strategies. However, a multi degree of freedom robot arm has many parameters, and the process of determining them is challenging. Much research has been done in this area and experimental methods have been applied on several robot arms. To our knowledge, there is currently no set of inertial parameters, either by modelling or by estimation, available for the CRS A460/A465 arm, a popular laboratory table top robot. In this paper we review and compare a number of methods for dynamic parameter identification and for generating trajectories suitable for estimating the identifiable dynamic parameters of a given robot. We then present a step by step process for dynamic parameter identification of a serial manipulator, and demonstrate this process by experimentally identifying the dynamic parameters of the CRS A460 robot.

IROS Conference 2007 Conference Paper

Interactive topology formation of linguistic space and motion space

• Wataru Takano
• Dana Kulic
• Yoshihiko Nakamura

hierarchical model incorporating motion patterns, proto symbols and words is proposed. The proto symbols abstract motion patterns, while the words are associated with the proto symbols stochastically. This paper describes the construction of a word space, where words are located in a multidimensional space based on dissimilarities among the words. The dissimilarity between two words can be calculated by using association probabilities that the words generate motion proto symbols. The word space encapsulates relations among the words such as similar or dissimilar pairs of words. The word space also allows motion recognition based on words. The validity of the constructed word space is demonstrated on a motion capture database. Moreover, the addition of the word associations is found to change the conventional proto symbol space so that the discrimination among the proto symbols is improved.

IROS Conference 2007 Conference Paper

Representability of human motions by factorial hidden Markov models

• Dana Kulic
• Wataru Takano
• Yoshihiko Nakamura

This paper describes an improved methodology for human motion recognition and imitation based on Factorial Hidden Markov Models (FHMM). Unlike conventional Hidden Markov Models (HMMs), FHMMs use a distributed state representation, which allows for more efficient representation of each time sequence. Once the FHMMs are trained with exemplar motion data, they can be used to generate sample trajectories for motion production, and produce significantly more accurate trajectories compared to single Hidden Markov chain models. Due to the additional information encoded in FHMMs models, FHMM models have a higher Kullback- Leibler distance compared to single Markov chain models, making it easier to distinguish between similar models. The efficacy of using FHMMs is tested on a database of human motions obtained through motion capture. The results show that FHMMs provide better generalization to new data when compared to conventional HMMs during motion recognition, as well as providing a better fit for generated data.

IROS Conference 2005 Conference Paper

Anxiety detection during human-robot interaction

• Dana Kulic
• Elizabeth A. Croft

This paper describes an experiment to determine the feasibility of using physiological signals to determine the human response to robot motions during direct human-robot interaction. A robot manipulator is used to generate common interaction motions, and human subjects are asked to report their response to the motions. The human physiological response is also measured. Motion paths are generated using a classic potential field planner and a safe motion planner, which minimizes the potential collision force along the path. A fuzzy inference engine is developed to estimate the human response based on the physiological measures. Results show that emotional arousal can be detected using physiological signals and the inference engine. Comparison of initial results between the two planners shows that subjects report less anxiety and surprise with the safe planner for high planner speeds.

ICRA Conference 2004 Conference Paper

Safe Planning for Human-Robot Interaction

• Dana Kulic
• Elizabeth A. Croft

A two stage planning approach is proposed to address issues of potentially conflicting planning criteria. This paper presents a strategy for improving the safety of human-robot interaction for articulated robots by minimizing a danger criterion during the planning stage. Two formulations of the danger criterion are proposed: a criterion assuming independent factors, and a criterion assuming mutually dependent factors. Simulations of the proposed planning strategy are presented for both 2D and 3D robots.