Author name cluster

Takamitsu Matsubara

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

32 papers

2 author rows

ICRA Conference 2025 Conference Paper

Feasibility-Aware Imitation Learning from Observations Through a Hand-Mounted Demonstration Interface

Kei Takahashi
Hikaru Sasaki
Takamitsu Matsubara

Imitation learning through a demonstration interface is expected to learn policies for robot automation from intuitive human demonstrations. However, due to the differences in human and robot movement characteristics, a human expert might unintentionally demonstrate an action that the robot cannot execute. We propose feasibility-aware behavior cloning from observation (FABCO). In the FABCO framework, the feasibility of each demonstration is assessed using the robot's pre-trained forward and inverse dynamics models. This feasibility information is provided as visual feedback to the demonstrators, encouraging them to refine their demonstrations. During policy learning, estimated feasibility serves as a weight for the demonstration data, improving both the data efficiency and the robustness of the learned policy. We experimentally validated FABCO's effectiveness by applying it to a pipette insertion task involving a pipette and a vial. Four participants assessed the impact of the feasibility feedback and the weighted policy learning in FABCO. Additionally, we used the NASA Task Load Index (NASA-TLX) to evaluate the workload induced by demonstrations with visual feedback.

IROS Conference 2025 Conference Paper

ICCO: Learning an Instruction-conditioned Coordinator for Language-guided Task-aligned Multi-robot Control

Yoshiki Yano
Kazuki Shibata
Maarten Kokshoorn
Takamitsu Matsubara

Recent advances in Large Language Models (LLMs) have permitted the development of language-guided multi-robot systems, which allow robots to execute tasks based on natural language instructions. However, achieving effective coordination in distributed multi-agent environments remains challenging due to (1) misalignment between instructions and task requirements and (2) inconsistency in robot behaviors when they independently interpret ambiguous instructions. To address these challenges, we propose Instruction-Conditioned Coordinator (ICCO), a Multi-Agent Reinforcement Learning (MARL) framework designed to enhance coordination in language-guided multi-robot systems. ICCO consists of a Coordinator agent and multiple Local Agents, where the Coordinator generates Task-Aligned and Consistent Instructions (TACI) by integrating language instructions with environmental states, ensuring task alignment and behavioral consistency. The Coordinator and Local Agents are jointly trained to optimize a reward function that balances task efficiency and instruction following. A Consistency Enhancement Term is added to the learning objective to maximize mutual information between instructions and robot behaviors, further improving coordination. Simulation and real-world experiments validate the effectiveness of ICCO in achieving language-guided task-aligned multi-robot control. The demonstration can be found at https://yanoyoshiki.github.io/ICCO/.

IROS Conference 2024 Conference Paper

Domain Randomization-free Sim-to-Real: An Attention-Augmented Memory Approach for Robotic Tasks

Jia Qu
Shun Otsubo
Tomoya Yamanokuchi
Takamitsu Matsubara
Shotaro Miwa

The sim-to-real gap, a long-standing challenge in the field of robotics, has garnered significant attention. Essentially, it is important to learn robust representation models that can be seamlessly applied in both simulation and real world. Traditional approaches like domain randomization have demonstrated success in zero-short setting, by creating representations that are resilient and adaptable through the augmentation of diversity within simulations. However, they suffer from the need for extensive training across a range of parameter variances, and dependency on heuristic approaches. In this work, we present a novel reinforcement learning architecture named Soft Attention-Augmented Actor-Critic (Soft3AC) for sim-to-real robotic tasks without the need for heuristic domain randomization. Our approach achieves the learning of semantically task-relevant feature representations that exhibit resilience against appearance gaps. This is realized by employing an architectural design that separates current perceptions from historical perceptions in memory, fostering abstract spatial-temporal understanding. Simultaneously, the introduction of an attention mechanism enables a more contextual processing. We validated our method through conducting a valve rotation task with a robotic hand, under both sim-to-sim and sim-to-real conditions. The results indicate that our model adeptly bridges the appearance gap observed in sim-to-sim and sim-to-real transfers. Our method demonstrated its ability to be deployed directly into the real world in a domain randomization free zero-shot manner.

IROS Conference 2023 Conference Paper

Domains as Objectives: Multi-Domain Reinforcement Learning with Convex-Coverage Set Learning for Domain Uncertainty Awareness

Wendyam Eric Lionel Ilboudo
Taisuke Kobayashi
Takamitsu Matsubara

Domain randomization (DR) is a powerful framework that has allowed the transfer of policies from randomized domain (a. k. a. simulation) to real robots with little to no retraining requirement. However, because the policy has to perform well for many different domain conditions, DR tends to produce sub-optimal policies that can be too conservative on the target real system. This problem is further exacerbated the larger the randomized domain is. To tackle this issue, recent works have proposed to learn universal policies (UP) with domain knowledge such that they can adapt their behavior to each domain when paired with an online system identifier (OSI). However, in most applications, perfect identifications of the target domain can be impossible. In this paper, by drawing similarities between DR as a multi-domain reinforcement learning and multi-objective reinforcement learning (MORL), we propose to learn a UP over the convex coverage set borrowed from the MORL theory. Thanks to this, our method learns a UP that effectively captures different sub-domains of the uncertainty set and can therefore adapt its behavior based on an OSI uncertainty, unlocking the power of stochastic system identification with no retraining requirement. This pseudo-MORL framework also contains previous works in DR and robust reinforcement learning. We conduct simulations on Mujoco tasks and experiments on a real D'Claw robot, revealing the effectiveness of our domain-uncertainty-aware UP for sim-to-real transfer.

ICRA Conference 2022 Conference Paper

Disturbance-injected Robust Imitation Learning with Task Achievement

Hirotaka Tahara
Hikaru Sasaki
Hanbit Oh
Brendan Michael
Takamitsu Matsubara

Robust imitation learning using disturbance injections overcomes issues of limited variation in demonstrations. However, these methods assume demonstrations are optimal, and that policy stabilization can be learned via simple augmentations. In real-world scenarios, demonstrations are often of diverse-quality, and disturbance injection instead learns sub-optimal policies that fail to replicate desired behavior. To address this issue, this paper proposes a novel imitation learning framework that combines both policy robustification and optimal demonstration learning. Specifically, this combinatorial approach forces policy learning and disturbance injection optimization to focus on mainly learning from high task achievement demonstrations, while utilizing low achievement ones to decrease the number of samples needed. The effectiveness of the proposed method is verified through experiments using an excavation task in both simulations and a real robot, resulting in high-achieving policies that are more stable and robust to diverse-quality demonstrations. In addition, this method utilizes all of the weighted sub-optimal demonstrations without eliminating them, resulting in practical data efficiency benefits.

ICRA Conference 2022 Conference Paper

Gaussian Process Self-triggered Policy Search in Weakly Observable Environments

Hikaru Sasaki
Terushi Hirabayashi
Kaoru Kawabata
Takamitsu Matsubara

The environments of such large industrial machines as waste cranes in waste incineration plants are often weakly observable, where little information about the environ-mental state is contained in the observations due to technical difficulty or maintenance cost (e. g. , no sensors for observing the state of the garbage to be handled). Based on the findings that skilled operators in such environments choose predetermined control strategies (e. g. , grasping and scattering) and their durations based on sensor values, we propose a novel non-parametric policy search algorithm: Gaussian process self-triggered policy search (GPSTPS). GPSTPS has two types of control policies: action and duration. A gating mechanism either maintains the action selected by the action policy for the duration specified by the duration policy or updates the action and duration by passing new observations to the policy; therefore, it is categorized as self-triggered. GPSTPS simultaneously learns both policies by trial and error based on sparse GP priors and variational learning to maximize the return. To verify the performance of our proposed method, we conducted experiments on garbage-grasping-scattering task for a waste crane with weak observations using a simulation and a robotic waste crane system. As experimental results, the proposed method acquired suitable policies to determine the action and duration based on the garbage's characteristics.

ICRA Conference 2021 Conference Paper

Bayesian Disturbance Injection: Robust Imitation Learning of Flexible Policies

Hanbit Oh
Hikaru Sasaki
Brendan Michael
Takamitsu Matsubara

Scenarios requiring humans to choose from multiple seemingly optimal actions are commonplace, however standard imitation learning often fails to capture this behavior. Instead, an over-reliance on replicating expert actions induces inflexible and unstable policies, leading to poor generalizability in an application. To address the problem, this paper presents the first imitation learning framework that incorporates Bayesian variational inference for learning flexible nonparametric multi-action policies, while simultaneously robustifying the policies against sources of error, by introducing and optimizing disturbances to create a richer demonstration dataset. This combinatorial approach forces the policy to adapt to challenging situations, enabling stable multi-action policies to be learned efficiently. The effectiveness of our proposed method is evaluated through simulations and real-robot experiments for a table-sweep task using the UR3 6-DOF robotic arm. Results show that, through improved flexibility and robustness, the learning performance and control safety are better than comparison methods.

ICRA Conference 2021 Conference Paper

Deep reinforcement learning of event-triggered communication and control for multi-agent cooperative transport

Kazuki Shibata
Tomohiko Jimbo
Takamitsu Matsubara

In this paper, we explore a multi-agent reinforcement learning approach to address the design problem of communication and control strategies for multi-agent cooperative transport. Typical end-to-end deep neural network policies may be insufficient for covering communication and control; these methods cannot decide the timing of communication and can only work with fixed-rate communications. Therefore, our framework exploits event-triggered architecture, namely, a feedback controller that computes the communication input and a triggering mechanism that determines when the input has to be updated again. Such event-triggered control policies are efficiently optimized using a multi-agent deep deterministic policy gradient. We confirmed that our approach could balance the transport performance and communication savings through numerical simulations.

IROS Conference 2021 Conference Paper

Learning Robotic Contact Juggling

Kazutoshi Tanaka
Masashi Hamaya
Devwrat Joshi
Felix von Drigalski
Ryo Yonetani
Takamitsu Matsubara
Yoshihisa Ijiri

Robotic contact juggling is a challenging task in which robots must control the movement of a ball rapidly and indirectly without holding it while keeping the ball in and sometimes out of contact with the robot’s body. In this work, we address the problem of learning such robotic contact juggling from trial and error via model-based reinforcement learning (MBRL). The key insight is that complex robot-ball interactions of the contact juggling actually consist of a small set of simple dynamics that each corresponds to a distinct interaction "primitive" such as touching and releasing the ball. Accordingly, we develop a tailored MBRL method that incrementally fits a set of simple dynamics models to the movements of a robot and a ball while also learning a switching model that can select a proper dynamics model depending on the current state and action. The learned model can then be used in an MBRL framework to seek optimal juggling control. We demonstrated the effectiveness of our approach on a simulator of contact juggling performed by a robotic arm.

ICRA Conference 2020 Conference Paper

Contact-based in-hand pose estimation using Bayesian state estimation and particle filtering

Felix von Drigalski
Shohei Taniguchi
Robert Lee
Takamitsu Matsubara
Masashi Hamaya
Kazutoshi Tanaka
Yoshihisa Ijiri

In industrial assembly tasks, the position of an object grasped by the robot has to be known with high precision in order to insert or place it. In real applications, this problem is commonly solved by jigs that are specially produced for each part. However, they significantly limit flexibility and are prohibitive when the target parts change often, so a flexible method to localize parts with high accuracy after grasping is desired. To solve this problem, we propose a method that can estimate the position of an object in the robot's hand to sub-millimeter precision, and can improve its estimate incrementally, using only minimal calibration and a force sensor. Our method is applicable to any robotic gripper and any rigid object that the gripper can hold, and requires only a force sensor. We demonstrate that the method can determine the position of an object to a precision of under 1 mm without using any part-specific jigs or equipment.

ICRA Conference 2020 Conference Paper

Dynamic Actor-Advisor Programming for Scalable Safe Reinforcement Learning

Lingwei Zhu
Yunduan Cui
Takamitsu Matsubara

Real-world robots have complex strict constraints. Therefore, safe reinforcement learning algorithms that can simultaneously minimize the total cost and the risk of constraint violation are crucial. However, almost no algorithms exist that can scale to high-dimensional systems to the best of our knowledge. In this paper, we propose Dynamic Actor-Advisor Programming (DAAP), as an algorithm for sample-efficient and scalable safe reinforcement learning. DAAP employs two control policies, actor and advisor. They are updated to minimize total cost and risk of constraint violation intertwiningly and smoothly towards each other's direction by using the other as the baseline policy in the Kullback-Leibler divergence of Dynamic Policy Programming framework. We demonstrate the scalability and sample efficiency of DAAP through its application on simulated robot arm control tasks with performance comparisons to baselines.

IROS Conference 2020 Conference Paper

Learning Soft Robotic Assembly Strategies from Successful and Failed Demonstrations

Masashi Hamaya
Felix von Drigalski
Takamitsu Matsubara
Kazutoshi Tanaka
Robert Lee
Chisato Nakashima
Yoshiya Shibata
Yoshihisa Ijiri

Physically soft robots are promising for robotic assembly tasks as they allow stable contacts with the environment. In this study, we propose a novel learning system for soft robotic assembly strategies. We formulate this problem as a reinforcement learning task and design the reward function from human demonstrations. Our key insight is that the failed demonstrations can be used as constraints to avoid failed behaviors. To this end, we developed a teaching device with which humans can intuitively provide various demonstrations. Moreover, we leverage Physically-Consistent Gaussian Mixture Models to clearly assign Gaussian components to the successful and failed trials. We then create the reference trajectories via Gaussian Mixture Regressions, which fit the successful demonstrations while considering the failed ones. Finally, we apply a sample- efficient deep model-based reinforcement learning method to obtain robust strategies with a few interactions. To validate our method, we developed a real-robot experimental system composed of a rigid collaborative robot arm with a compliant wrist and the teaching device. Our results demonstrated that our method learned the assembly strategies with a higher success rate than when using only successful demonstrations.

ICRA Conference 2020 Conference Paper

Sample-and-computation-efficient Probabilistic Model Predictive Control with Random Features

Cheng-Yu Kuo
Yunduan Cui
Takamitsu Matsubara

Gaussian processes (GPs) based Reinforcement Learning (RL) methods with Model Predictive Control (MPC) have demonstrated their excellent sample efficiency. However, since the computational cost of GPs largely depends on the training sample size, learning an accurate dynamics using GPs result in low control frequency in MPC. To alleviate this trade-off and achieve a sample-and-computation-efficient nature, we propose a novel model-based RL method with MPC. Our approach employs a linear Gaussian model with randomized features using the Fastfood as an approximated GP dynamics. Then, we derive an analytic moment-matching scheme in state prediction with the model and uncertain inputs. As a result, the computational cost of the MPC in our RL method does not depend on the training sample size and can improve the control frequency over previous methods. Through experiments with simulated and real robot control tasks, the sample efficiency, as well as the computation efficiency of our model-based RL method, are demonstrated.

ICRA Conference 2019 Conference Paper

Exploiting Human and Robot Muscle Synergies for Human-in-the-loop Optimization of EMG-based Assistive Strategies

Masashi Hamaya
Takamitsu Matsubara
Jun-ichiro Furukawa
Yuting Sun
Satoshi Yagi
Tatsuya Teramae
Tomoyuki Noda
Jun Morimoto

In this study, we propose a novel human-in-the-loop optimization approach for exoskeleton robot control. We develop a method to optimize widely-used Electromyography (EMG)-based assistive strategies. If we use multiple EMG channels to control multi-DoF robots, optimization process becomes complex and requires a large amount of data. To make the optimization tractable, we exploit the synergies both of the human muscles and artificial muscles of the exoskeleton robots to reduce the number of parameters of the assistive strategies. We show that we can extract the synergies not only from the user's muscle activities but from pneumatic artificial muscle (PAMs) contractions of the exoskeleton robot. Then, we adopt a Bayesian optimization method to acquire the parameters for assisting human movements by iteratively identifying the user's preferences of the assistive strategies. We conducted experiments to evaluate our proposed method with a PAMs-driven upper-limb exoskeleton robot. Our method successfully learned assistive strategies from the human-in-theloop optimization with a practicable number of interactions.

ICRA Conference 2019 Conference Paper

Multimodal Policy Search using Overlapping Mixtures of Sparse Gaussian Process Prior

Hikaru Sasaki
Takamitsu Matsubara

In this paper, we present a novel policy search reinforcement learning algorithm that can deal with multimodality in control policies based on Gaussian processes. Our approach employs Overlapping Mixtures of Gaussian Processes (OMGPs) for a control policy, in which all the GPs in the mixture are global and overlapped in the input space. We first extend the OMGPs by combing sparse pseudo-input GPs as OMSGPs to reduce its computational cost of learning and prediction suitable for policy search. Then, we derive a novel multimodal policy search algorithm based on variational Bayesian inference by placing the OMSGPs as the prior of the multimodal control policy. To validate the effectiveness of our algorithm, we applied it to two typical robotic tasks in simulation: 1) object grasping and 2) table-sweep tasks since they both require the multimodality in the optimal policies. Simulation results demonstrate that our algorithm can efficiently learn multimodal policies even with high dimensional observations.

ICRA Conference 2019 Conference Paper

Probabilistic Active Filtering for Object Search in Clutter

James Poon
Yunduan Cui
Junichiro Ooga
Akihito Ogawa
Takamitsu Matsubara

This paper proposes a probabilistic approach for object search in clutter. Due to heavy occlusions, it is vital for an agent to be able to gradually reduce uncertainty in observations of the objects in its workspace by systematically rearranging them. Probabilistic methodologies present a promising sample-efficient alternative to handle the massively complex state-action space that inherently comes with this problem, avoiding the need for both exhaustive training samples and the accompanying heuristics for traversing a large-scale model during runtime. We approach the object search problem by extending a Gaussian Process active filtering strategy with an additional model for capturing state dynamics as the objects are moved over the course of the activity. This allows viable models to be built upon relatively scarce training data, while the complexity of the action space is also reduced by shifting objects over relatively short distances. Validation in both simulation and with a real Baxter robot with a limited number of training samples demonstrates the efficacy of the proposed approach.

IROS Conference 2019 Conference Paper

Reinforcement Learning Boat Autopilot: A Sample-efficient and Model Predictive Control based Approach

Yunduan Cui
Shigeki Osaki
Takamitsu Matsubara

In this research we focus on developing a reinforcement learning system for a challenging task: autonomous control of a real-sized boat, with difficulties arising from large uncertainties in the challenging ocean environment and the extremely high cost of exploring and sampling with a real boat. To this end, we explore a novel Gaussian processes (GP) based reinforcement learning approach that combines sample-efficient model-based reinforcement learning and model predictive control (MPC). Our approach, sample-efficient probabilistic model predictive control (SPMPC), iteratively learns a Gaussian process dynamics model and uses it to efficiently update control signals within the MPC closed control loop. A system using SPMPC is built to efficiently learn an autopilot task. After investigating its performance in a simulation modeled upon real boat driving data, the proposed system successfully learns to drive a real-sized boat equipped with a single engine and sensors measuring GPS, speed, direction, and wind in an autopilot task without human demonstration.

IROS Conference 2017 Conference Paper

Deep dynamic policy programming for robot control with raw images

Yoshihisa Tsurumine
Yunduan Cui
Eiji Uchibe
Takamitsu Matsubara

Deep reinforcement learning has drawn much attention in robot control since it enables agents to learn control policies from very high dimensional states such as raw images. On the other hand, its dependency upon the availability of a significant quantity of training samples and its fragility in learning makes it difficult to apply for real world robot tasks. To alleviate these issues we propose Deep Dynamic Policy Programming (DDPP), which combines the sample efficiency and smooth policy updates of dynamic policy programming with the contemporary deep reinforcement learning framework. The effectiveness of the proposed method is first demonstrated in a simulation of the robot arm control problem, with comparison to Deep Q-Networks. As validation on a real robot system, DDPP also successfully learned the flipping of a handkerchief with a NEXTAGE humanoid robot using a reduced number of learning samples, whereas Deep Q-Networks failed to learn the task.

ICRA Conference 2017 Conference Paper

Learning task-parametrized assistive strategies for exoskeleton robots by multi-task reinforcement learning

Masashi Hamaya
Takamitsu Matsubara
Tomoyuki Noda
Tatsuya Teramae
Jun Morimoto

Recent studies suggest that reinforcement learning has great potential for generating assistive strategies in exoskeletons through physical interactions between a user and a robot. Previous methods focused on a task-specific assistive strategy, where for every single task (situation/context), the user needs to interact with a robot to learn an appropriate assistive strategy. Therefore, the learned strategies cannot be generalized for a new task. Since the sampling cost is expensive for such human-in-the-loop systems as exoskeletons, generalization must be enabled. In this paper, we propose to learn task-parametrized assistive strategies for exoskeleton robots. Our method employs an assistive strategy, which depends on the task parameter and the state variable, that can be learned from multiple sets of human-robot interaction data across different tasks and generalized even for an unseen task, given the task parameter without additional learning. To alleviate the user's burden in the learning process across multiple tasks, we exploit a data-efficient multi-task reinforcement learning framework. To verify the effectiveness of our method, we developed an experimental platform with an exoskeleton robot. We conducted a series of experiments whose experimental results show that our method can learn such a task-parametrized assistive strategy and be generalized for unseen tasks to reduce the user's electromyography signals (EMGs) during tasks.

ICRA Conference 2017 Conference Paper

Local driving assistance from demonstration for mobility aids

James Poon
Yunduan Cui
Jaime Valls Miró
Takamitsu Matsubara
Kenji Sugimoto

Active assistive mobility systems are largely limited to a-priori mapped environments, whereas their reactive assistive counterparts are in general location independent and focus on the provision of collision avoidance in the immediate space surrounding the platform. This paper presents a framework capable of providing active short-term navigation, combining the intelligence of active assistance with the freedom of location independence. Demonstration data from an able expert while driving the mobility aid in a standard indoor setting is used off-line to learn reference behavioral models of navigation given perceptual information from the platform surroundings and the input controls exerted by the user while navigating. These serve as the foundation for on-line probabilistic short-term destination inference using the instantaneously available data from the user and on-board sensors. This is coupled with a real-time stochastic optimal path generation able to exploit the same short term demonstration paths from the expert with the belief they capture both the driver's awareness of the platform's physical geometry and appropriate behaviors for their surroundings. Experimental results with users of varying proficiency in a setting unvisited in training data show promise in using the framework in assisting users experiencing difficulty in safe power mobility aid use.

IROS Conference 2017 Conference Paper

User-robot collaborative excitation for PAM model identification in exoskeleton robots

Masashi Hamaya
Takamitsu Matsubara
Tomoyuki Noda
Tatsuya Teramae
Jun Morimoto

Pneumatic Artificial Muscle (PAM) actuators have been used as exoskeletons because of their inherited compliance and high power-weight ratio. However, creating accurate models remains difficult mainly due to the compliance issue; the model can be changed by the force applied by the user. Therefore, both user and robot actions need to be considered for sufficient excitation of PAMs that are equipped in exoskeleton robots, unlike typical rigid actuators that can only be sufficiently excited by robot actions. In this paper, we propose a user-robot collaborative excitation approach for PAM model identification as an active learning framework for sequentially collecting data by deriving and executing optimal user and robot actions at each step with Gaussian processes. The optimal actions, which are executed by the robot, are displayed on a monitor that enables the user to execute them. We conducted experiments using a powered elbow exoskeleton with a PAM actuator. Experimental results show that our method can more efficiently identify the PAM model than a standard model identification method that does not use any data acquired through user-robot collaboration.

ICRA Conference 2016 Conference Paper

Learning assistive strategies from a few user-robot interactions: Model-based reinforcement learning approach

Masashi Hamaya
Takamitsu Matsubara
Tomoyuki Noda
Tatsuya Teramae
Jun Morimoto

Designing an assistive strategy for exoskeletons is a key ingredient in movement assistance and rehabilitation. While several approaches have been explored, most studies are based on mechanical models of the human user, i. e. , rigid-body dynamics or Center of Mass (CoM)-Zero Moment Point (ZMP) inverted pendulum moECenter of Massdel, or only focus on periodic movements with using oscillator models. On the other hand, the interactions between the user and the robot are often not considered explicitly because of its difficulty in modeling. In this paper, we propose to learn the assistive strategies directly from interactions between the user and the robot. We formulate the learning problem of assistive strategies as a policy search problem. To alleviate heavy burdens to the user for data acquisition, we exploit a data-efficient model-based reinforcement learning framework. To validate the effectiveness of our approach, an experimental platform composed of a real subject, an electromyography (EMG)-measurement system, and a simulated robot arm is developed. Then, a learning experiment with the assistive control task of the robot arm is conducted. As a result, proper assistive strategies that can achieve the robot control task and reduce EMG signals of the user are acquired only by 30 seconds interactions.

UAI Conference 2014 Conference Paper

Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models

Takamitsu Matsubara
Vicenç Gómez
Hilbert J. Kappen

Kullback Leibler (KL) control problems allow for efficient computation of optimal control by solving a principal eigenvector problem. However, direct applicability of such framework to continuous state-action systems is limited. In this paper, we propose to embed a KL control problem in a probabilistic graphical model where observed variables correspond to the continuous (possibly high-dimensional) state of the system and latent variables correspond to a discrete (low-dimensional) representation of the state amenable for KL control computation. We present two examples of this approach. The first one uses standard hidden Markov models (HMMs) and computes exact optimal control, but is only applicable to lowdimensional systems. The second one uses factorial HMMs, it is scalable to higher dimensional problems, but control computation is approximate. We illustrate both examples in several robot motor control tasks.

IROS Conference 2014 Conference Paper

Object manifold learning with action features for active tactile object recognition

Daisuke Tanaka
Takamitsu Matsubara
Kentaro Ichien
Kenji Sugimoto

In this paper, we consider an object recognition problem based on tactile information using a robot hand. The robot performs an exploratory action to the object to obtain the tactile information, however, poorly designed actions may not be sufficiently informative. In contrast, if we could collect sample data by sequentially performing informative actions, i. e. , active learning, the required time would be drastically reduced. To this end, we propose a novel approach for active tactile object recognition. Our approach combines both an active learning scheme and a nonlinear dimensionality reduction method. We first extracts the object manifold, each coordinate of which represents an object, from tactile sensor data and action features using Gaussian Process Latent Variable Models. At the same time, a probabilistic model of the observed data related to the action and the object are learned. Then, with the learned model, optimally-informative exploratory actions can be computed sequentially, and performed to efficiently collect the data for recognition. We show experimental results that verify the effectiveness of our proposed method with synthetic data and a real robot.

IROS Conference 2012 Conference Paper

Full-body exoskeleton robot control for walking assistance by style-phase adaptive pattern generation

Takamitsu Matsubara
Akimasa Uchikata
Jun Morimoto

We propose an adaptive walking assistance strategy to control an exoskeleton robot. In our proposed framework, we explicitly consider the following: 1) the diversity of user motions (style) and 2) the interactions among a user, a robot, and an environment. To spatially coordinate a wide variety of user motions and robot behaviors, we estimated style parameters from observed user movements. To temporally coordinate the interactions among the user, the robot, and the environment, we synchronized the phases of these three systems with a coupled oscillator model. The estimated style parameters and the phase of the user motion can be used to predict future user movements. We investigated how movement prediction and phase synchronization can be beneficial to control an exoskeleton robot. To evaluate our adaptive walking assistance strategy, we developed simulated user and exoskeleton models. The physical interactions among the user, the exoskeleton, and the ground models are introduced in the simulated system. We show that the necessary torque for the user walking movement was reduced around 40% by using our proposed method to control the exoskeleton model.

ICRA Conference 2012 Conference Paper

Spatio-temporal synchronization of periodic movements by style-phase adaptation: Application to biped walking

Takamitsu Matsubara
Akimasa Uchikata
Jun Morimoto

In this paper, we propose a framework for generating coordinated periodic movements of robotic systems with external inputs. We developed an adaptive pattern generator model that is composed of a two-factor observation model with a style parameter and phase dynamics with a phase variable. The style parameter controls the spatial patterns of the generated trajectories, and the phase variable controls its temporal profiles. To validate the effectiveness of our proposed method, we applied it to a simulated humanoid model to perform biped walking behaviors coordinated with observed walking patterns and the environment. The robot successfully performed stable biped walking behaviors even when the style of the observed walking pattern and the period were suddenly changed.

IROS Conference 2011 Conference Paper

XoR: Hybrid drive exoskeleton robot that can balance

Sang-Ho Hyon
Jun Morimoto
Takamitsu Matsubara
Tomoyuki Noda
Mitsuo Kawato

We propose a novel exoskeleton robot prototype aimed at a brain-machine interface and rehabilitation for postural control for elderly people, people with spinal cord injury, stroke patients, and others with similar needs. By arranging pneumatic muscles with electric motors in a optimal way, one can achieve both weight-reduction and torque-controllability. Its anthropomorphic design and torque-controllability enable users to implement and test various rehabilitation/compensation programs consistent with human motor control and learning mechanism. Hybrid drive itself is not new, but its specialized application to lightweight exoskeleton is novel. This paper reports the design and development of the robot, particularly addressing a hybrid drive for load-bearing tasks such as standing and postural maintenance. The experimental data as well as the attached videos demonstrate the effectiveness of the proposed system.

IROS Conference 2010 Conference Paper

Learning Stylistic Dynamic Movement Primitives from multiple demonstrations

Takamitsu Matsubara
Sang-Ho Hyon
Jun Morimoto

In this paper, we propose a novel concept of movement primitives called Stylistic Dynamic Movement Primitives (SDMPs) for motor learning and control in humanoid robotics. In the SDMPs, a diversity of styles in human motion observed through multiple demonstrations can be compactly encoded in a movement primitive, and this allows style manipulation of motion sequences generated from the movement primitive by a control variable called a style parameter. Focusing on discrete movements, a model of the SDMPs is presented as an extension of Dynamic Movement Primitives (DMPs) proposed by Ijspeert et al. . A novel learning procedure of the SDMPs from multiple demonstrations, including a diversity of motion styles, is also described. We present two practical applications of the SDMPs, i. e. , stylistic table tennis swings and obstacle avoidance with an anthropomorphic manipulator.

ICRA Conference 2010 Conference Paper

Optimal Feedback Control for anthropomorphic manipulators

Djordje Mitrovic
Sho Nagashima
Stefan Klanke
Takamitsu Matsubara
Sethu Vijayakumar

We study target reaching tasks of redundant anthropomorphic manipulators under the premise of minimal energy consumption and compliance during motion. We formulate this motor control problem in the framework of Optimal Feedback Control (OFC) by introducing a specific cost function that accounts for the physical constraints of the controlled plant. Using an approximative computational optimal control method we can optimally control a high-dimensional anthropomorphic robot without having to specify an explicit inverse kinematics, inverse dynamics or feedback control law. We highlight the benefits of this biologically plausible motor control strategy over traditional (open loop) optimal controllers: The presented approach proves to be significantly more energy efficient and compliant, while being accurate with respect to the task at hand. These properties are crucial for the control of mobile anthropomorphic robots, that are designed to interact safely in a human environment. To the best of our knowledge this is the first OFC implementation on a high-dimensional (redundant) manipulator.

ICRA Conference 2007 Conference Paper

Learning to acquire whole-body humanoid CoM movements to achieve dynamic tasks

Takamitsu Matsubara
Jun Morimoto
Jun Nakanishi
Sang-Ho Hyon
Joshua G. Hale
Gordon Cheng

This paper presents a novel approach to acquire dynamic whole-body movements on humanoid robots focused on learning a control policy for the center of mass. A policy-gradient method is used to acquire a CoM movement as a control policy for achieving a desired dynamic task. A CoM-Jacobian-based redundancy resolution is then used to compute angular velocities for all joints in order to achieve a whole-body movement consistent with the CoM movement acquired through learning. To demonstrate the effectiveness of our method, we apply it in simulation to the learning of a strong punching movement on the Fujitsu humanoid robot, Hoap-2.

AAAI Conference 2005 Conference Paper

Learning CPG Sensory Feedback with Policy Gradient for Biped Locomotion for a Full-Body Humanoid

Gen Endo
Takamitsu Matsubara

This paper describes a learning framework for a central pattern generator based biped locomotion controller using a policy gradient method. Our goals in this study are to achieve biped walking with a 3D hardware humanoid, and to develop an efficient learning algorithm with CPG by reducing the dimensionality of the state space used for learning. We demonstrate that an appropriate feedback controller can be acquired within a thousand trials by numerical simulations and the obtained controller in numerical simulation achieves stable walking with a physical robot in the real world. Numerical simulations and hardware experiments evaluated walking velocity and stability. Furthermore, we present the possibility of an additional online learning using a hardware robot to improve the controller within 200 iterations.

ICRA Conference 2005 Conference Paper

Learning Sensory Feedback to CPG with Policy Gradient for Biped Locomotion

Takamitsu Matsubara
Jun Morimoto
Jun Nakanishi
Masa-aki Sato
Kenji Doya

This paper proposes a learning framework for a CPG-based biped locomotion controller using a policy gradient method. Our goal in this study is to develop an efficient learning algorithm by reducing the dimensionality of the state space used for learning. We demonstrate that an appropriate feedback controller in the CPG-based controller can be acquired using the proposed method within a few thousand trials by numerical simulations. Furthermore, we implement the learned controller on the physical biped robot to experimentally show that the learned controller successfully works in the real environment.