Kenji Doya

AAAI Conference 2025 Short Paper

Training Recurrent Neural Networks with Inherent Missing Data for Wearable Device Applications (Student Abstract)

• Sutashu Tomonaga
• Haruo Mizutani
• Kenji Doya

Wearable devices are transforming healthcare by providing continuous, real-time physiological data for monitoring and analysis. However, data often suffer from noise and significant missing values due to operational constraints and user compliance. Traditional approaches address these issues through data imputation during pre-processing, introducing biases and inaccuracies. We propose a novel method enabling Recurrent Neural Networks (RNNs) to inherently handle missing data without imputation. By implementing teacher-forcing during Backpropagation Through Time (BPTT) when data are available and switching to autonomous mode otherwise, our approach leverages RNNs' dynamics to model physiological signals accurately. We demonstrate our method's effectiveness using the Lorenz 63 system as a surrogate dataset, achieving robust reconstructions with 80% missing data.

YNIMG Journal 2024 Journal Article

A whole-brain analysis of functional connectivity and immediate early gene expression reveals functional network shifts after operant learning

• Kazumi Kasahara
• Keigo Hikishima
• Mariko Nakata
• Tomokazu Tsurugizawa
• Noriyuki Higo
• Kenji Doya

Previous studies of operant learning have addressed neuronal activities and network changes in specific brain areas, such as the striatum, sensorimotor cortex, prefrontal/orbitofrontal cortices, and hippocampus. However, how changes in the whole-brain network are caused by cellular-level changes remains unclear. We, therefore, combined resting-state functional magnetic resonance imaging (rsfMRI) and whole-brain immunohistochemical analysis of early growth response 1 (EGR1), a marker of neural plasticity, to elucidate the temporal and spatial changes in functional networks and underlying cellular processes during operant learning. We used an 11.7-Tesla MRI scanner and whole-brain immunohistochemical analysis of EGR1 in mice during the early and late stages of operant learning. In the operant training, mice received a reward when they pressed left and right buttons alternately, and were punished with a bright light when they made a mistake. A group of mice (n = 22) underwent the first rsfMRI acquisition before behavioral sessions, the second acquisition after 3 training-session-days (early stage), and the third after 21 training-session-days (late stage). Another group of mice (n = 40) was subjected to histological analysis 15 min after the early or late stages of behavioral sessions. Functional connectivity increased between the limbic areas and thalamus or auditory cortex after the early stage of training, and between the motor cortex, sensory cortex, and striatum after the late stage of training. The density of EGR1-immunopositive cells in the motor and sensory cortices increased in both the early and late stages of training, whereas the density in the amygdala increased only in the early stage of training. The subcortical networks centered around the limbic areas that emerged in the early stage have been implicated in rewards, pleasures, and fears. The connectivities between the motor cortex, somatosensory cortex, and striatum that consolidated in the late stage have been implicated in motor learning. Our multimodal longitudinal study successfully revealed temporal shifts in brain regions involved in behavioral learning together with the underlying cellular-level plasticity between these regions. Our study represents a first step towards establishing a new experimental paradigm that combines rsfMRI and immunohistochemistry to link macroscopic and microscopic mechanisms involved in learning.

TMLR Journal 2023 Journal Article

Numerical Data Imputation for Multimodal Data Sets: A Probabilistic Nearest-Neighbor Kernel Density Approach

• Florian Lalande
• Kenji Doya

Numerical data imputation algorithms replace missing values by estimates to leverage incomplete data sets. Current imputation methods seek to minimize the error between the unobserved ground truth and the imputed values. But this strategy can create artifacts leading to poor imputation in the presence of multimodal or complex distributions. To tackle this problem, we introduce the $k$NN$\times$KDE algorithm: a data imputation method combining nearest neighbor estimation ($k$NN) and density estimation with Gaussian kernels (KDE). We compare our method with previous data imputation methods using artificial and real-world data with different data missing scenarios and various data missing rates, and show that our method can cope with complex original data structure, yields lower data imputation errors, and provides probabilistic estimates with higher likelihood than current methods. We release the code in open-source for the community.

ICLR Conference 2022 Conference Paper

Variational oracle guiding for reinforcement learning

• Dongqi Han
• Tadashi Kozuno
• Xufang Luo
• Zhao-Yun Chen
• Kenji Doya
• Yuqing Yang 0001
• Dongsheng Li 0002

How to make intelligent decisions is a central problem in machine learning and artificial intelligence. Despite recent successes of deep reinforcement learning (RL) in various decision making problems, an important but under-explored aspect is how to leverage oracle observation (the information that is invisible during online decision making, but is available during offline training) to facilitate learning. For example, human experts will look at the replay after a Poker game, in which they can check the opponents' hands to improve their estimation of the opponents' hands from the visible information during playing. In this work, we study such problems based on Bayesian theory and derive an objective to leverage oracle observation in RL using variational methods. Our key contribution is to propose a general learning framework referred to as variational latent oracle guiding (VLOG) for DRL. VLOG is featured with preferable properties such as its robust and promising performance and its versatility to incorporate with any value-based DRL algorithm. We empirically demonstrate the effectiveness of VLOG in online and offline RL domains with tasks ranging from video games to a challenging tile-based game Mahjong. Furthermore, we publish the Mahjong environment and an offline RL dataset as a benchmark to facilitate future research on oracle guiding (https://github.com/Agony5757/mahjong).

YNIMG Journal 2020 Journal Article

Diffusion functional MRI reveals global brain network functional abnormalities driven by targeted local activity in a neuropsychiatric disease mouse model

• Yoshifumi Abe
• Norio Takata
• Yuki Sakai
• Hiro Taiyo Hamada
• Yuichi Hiraoka
• Tomomi Aida
• Kohichi Tanaka
• Denis Le Bihan

Diffusion functional magnetic resonance imaging (DfMRI) has been proposed as an alternative functional imaging method to detect brain activity without confounding hemodynamic effects. Here, taking advantage of this DfMRI feature, we investigated abnormalities of dynamic brain function in a neuropsychiatric disease mouse model (glial glutamate transporter-knockdown mice with obsessive–compulsive disorder [OCD]-related behavior). Our DfMRI approaches consisted of three analyses: resting state brain activity, functional connectivity, and propagation of neural information. We detected hyperactivation and biased connectivity across the cortico-striatal-thalamic circuitry, which is consistent with known blood oxygen-level dependent (BOLD)-fMRI patterns in OCD patients. In addition, we performed ignition-driven mean integration (IDMI) analysis, which combined activity and connectivity analyses, to evaluate neural propagation initiated from brain activation. This analysis revealed an unbalanced distribution of neural propagation initiated from intrinsic local activation to the global network, while these were not detected by the conventional method with BOLD-fMRI. This abnormal function detected by DfMRI was associated with OCD-related behavior. Together, our comprehensive DfMRI approaches can successfully provide information on dynamic brain function in normal and diseased brains.

ICLR Conference 2020 Conference Paper

Variational Recurrent Models for Solving Partially Observable Control Tasks

• Dongqi Han
• Kenji Doya
• Jun Tani

In partially observable (PO) environments, deep reinforcement learning (RL) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how to improve the policy. In this study, we propose an RL algorithm for solving PO tasks. Our method comprises two parts: a variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM. The proposed algorithm was tested in two types of PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization. Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner.

ICML Conference 2018 Conference Paper

PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos

• Paavo Parmas
• Carl E. Rasmussen
• Jan Peters 0001
• Kenji Doya

Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may be caused by a fundamental chaos-like nature of long chains of nonlinear computations. Not only do the magnitudes of the gradients become large, the direction of the gradients becomes essentially random. We show that reparameterization gradients suffer from the problem, while likelihood ratio gradients are robust. Using our insights, we develop a model-based policy search framework, Probabilistic Inference for Particle-Based Policy Search (PIPPS), which is easily extensible, and allows for almost arbitrary models and policies, while simultaneously matching the performance of previous data-efficient learning algorithms. Finally, we invent the total propagation algorithm, which efficiently computes a union over all pathwise derivative depths during a single backwards pass, automatically giving greater weight to estimators with lower variance, sometimes improving over reparameterization gradients by $10^6$ times.

RLDM Conference 2017 Conference Abstract

Fast Adaptation of Behavior to Changing Goals with a Gamma Ensemble

• Chris Reinke
• Eiji Uchibe
• Kenji Doya

Humans and artificial agents not only have to cope with changes in their environments, but also with changes in the goals that they want to achieve in those environments. For example, during foraging the goal could change from obtaining the most desirable food to securing food as rapidly as possible if there is time pressure. In reinforcement learning, the goal is defined by the reward function and how strongly rewards are discounted over time. If the goal changes, model-free value-based methods need to adapt their values to the new reward function or discounting strategy. This relearning is time-intensive and does not allow quick adaptation. We propose a new model-free algorithm, the Independent Gamma-Ensemble (IGE). It is inspired by the finding that the striatum has distinct regions to encode values computed by different discount factors. Similarly, the IGE has a set of distinct modules, which are Q-functions with a different discount factors. This allows the IGE to learn and store a repertoire of different behaviors. Furthermore, it allows information about the outcome of each behavior to be decoded, making it possible to choose the best behavior for a new goal without relearning values. In a task with changing goals, the IGE outperformed a classical Q-learning agent. The IGE is a step toward adaptive artificial agents that can cope with dynamic environments in which goals also change. Furthermore, the IGE provides a model for the potential function of the modular structure in the striatum. The striatum, which is involved in habit learning, may learn different habits in its distinct regions with different discounting factors. Depending on the context, which could be indicated by the stress level, for example, the most appropriate habit could be used without the need to relearn. This may mean that the striatum is able to learn and retain several habits for the same environment and to select them in a context-dependent manner.

RLDM Conference 2015 Conference Abstract

Inverse Reinforcement Learning with Density Ratio Estimation

• Eiji Uchibe
• Kenji Doya

This paper proposes a novel model-free inverse reinforcement learning method based on density ratio estimation under the framework of Dynamic Policy Programming. We show that the logarithm of the ratio between the optimal policy and the baseline policy is represented by the state-dependent reward and the value function. Our proposal is a two-stage learning procedure. At first, some density ratio estimation methods are used to estimate the density ratio of policies. Next, the least squares method with regularization is applied to estimate the state-dependent reward and the value function that satisfies the relation. Our method is data-efficient because those functions can be estimated from a set of state transitions while most of previous methods require a set of trajectories. In addition, we do not need to compute the integral such as evaluation of the partition function. The proposed method is applied into a real-robot navigation task and experimental results show its superiority over conventional methods. In particular, we show that the estimated reward and value functions are useful when forward reinforcement learning is performed with the theory of shaping reward. Poster M15*: Learning Dynamic Locomotion Skills for Terrains with Obstacles Xue Bin Peng*, University of British Columbia; Michiel van de Panne, University of British Columbia Using reinforcement learning to develop motor skills for articulated figures is challenging be- cause of state spaces and action spaces that are high dimensional and continuous. In this work, we learn control policies for dynamic gaits across terrains having sequences of gaps, walls, and steps. Results are demonstrated using physics-based simulations of a 21 link planar dog and a 7-link planar biped. Our ap- proach is characterized by a number of features, including: non-parametric representation of the value function and the control policy; value iteration using batched positive-TD updates; localized epsilon-greedy exploration; and an action parameterization that is tailored for the problem domain. In support of the non- parametric representation, we further optimize for a task-specific distance metric. The policies are computed offline using repeated iterations of epsilon-greedy exploration and value iteration. The final control policies then run in real time over novel terrains. We evaluate the impact of the key features of our skill learning pipeline on the resulting performance.

ICRA Conference 2014 Conference Paper

Combining learned controllers to achieve new goals based on linearly solvable MDPs

• Eiji Uchibe
• Kenji Doya

Learning complicated behaviors usually involves intensive manual tuning and expensive computational optimization because we have to solve a nonlinear Hamilton-Jacobi-Bellman (HJB) equation. Recently, Todorov proposed a class of the so-called Linearly solvable Markov Decision Process (LMDP) which converts a nonlinear HJB equation to a linear differential equation. Linearity of the simplified HJB equation allows us to apply superposition to derive a new composite controller from a set of learned primitive controllers. However, his method was a model-based approach and it was not evaluated in a real domain. This study proposes a model-free method which is similar to the Least Squares Temporal Difference (LSTD) learning. In this method, the exponentially transformed cost function can be regarded as the discount factor in LSTD. Our proposed method is applied to learning walking behaviors with the quadruped robot to evaluate in real robot experiments. The goal of each primitive task is to go to the specific target position in the environment and that of the composite task is to approach arbitrary region represented by the primitives' target positions. Experimental results show that the composite policy can be used as a good initial policy for the new task.

RLDM Conference 2013 Conference Abstract

Inverse Reinforcement Learning for Analysis of Human Behaviors

• Eiji Uchibe
• Shoko Ota
• Kenji Doya

Reinforcement Learning (RL) is a computational framework for investigating decision-making processes of both biological and artificial systems that can learn an optimal policy by interacting with an en- vironment. Previous studies assume that the reward/cost from the environment is identical to the reward/cost used by subjects, but it is not necessarily true because the learning process and results may differ between subjects. Although reward/cost strongly influences behavior, it is mainly explained by the difference of learning frameworks such as model-based and model-free RL, as well as by that of meta-parameters such as the learning rate, discount factor, and so on. Recently, several methods of Inverse Reinforcement Learning (IRL) have been proposed in the field of machine learning and robotics in order to implement imitation learning. IRL can infer the reward/cost function from the observed behaviors which are assumed to be opti- mal. As opposed to previous IRL studies, we use IRL as a tool to investigate the behaviors of human experts in order to find simple representation of reward/cost functions. We extend the IRL method proposed by Dvijotham and Todorov, in which the optimal state transition is parameterized by the value function. The value function is estimated from the observed behaviors by maximizing the log-likelihood. The gradient of the log-likelihood is numerically evaluated by the Metropolis-Hastings algorithm in which the uncontrolled dynamics is used as a proposed density. Next, the cost function is retrieved from the estimated value func- tion by minimizing the Bellman residual. To evaluate our method, we identify the cost function from human behaviors in performing a pole balancing task and elucidate difference of the cost functions among seven subjects and evaluate how the cost functions affect their performance.

YNIMG Journal 2012 Journal Article

Changing the structure of complex visuo-motor sequences selectively activates the fronto-parietal network

• V.S. Chandrasekhar Pammi
• K.P. Miyapuram
• Ahmed
• Kazuyuki Samejima
• Raju S. Bapi
• Kenji Doya

Previous brain imaging studies investigating motor sequence complexity have mainly examined the effect of increasing the length of pre-learned sequences. The novel contribution of this research is that we varied the structure of complex visuo-motor sequences along two different dimensions using mxn paradigm. The complexity of sequences is increased from 12 movements (organized as a 2×6 task) to 24 movements (organized as 4×6 and 2×12 tasks). Behavioral results indicate that although the success rate attained was similar across the two complex tasks (2×12 and 4×6), a greater decrease in response times was observed for the 2×12 compared to the 4×6 condition at an intermediate learning stage. This decrease is possibly related to successful chunking across sets in the 2×12 task. In line with this, we observed a selective activation of the fronto-parietal network. Shifts of activation were observed from the ventral to dorsal prefrontal, lateral to medial premotor and inferior to superior parietal cortex from the early to intermediate learning stage concomitant with an increase in hyperset length. We suggest that these selective activations and shifts in activity during complex sequence learning are possibly related to chunking of motor sequences.

NeurIPS Conference 2009 Conference Paper

A Generalized Natural Actor-Critic Algorithm

• Tetsuro Morimura
• Eiji Uchibe
• Junichiro Yoshimoto
• Kenji Doya

Policy gradient Reinforcement Learning (RL) algorithms have received much attention in seeking stochastic policies that maximize the average rewards. In addition, extensions based on the concept of the Natural Gradient (NG) show promising learning efficiency because these regard metrics for the task. Though there are two candidate metrics, Kakades Fisher Information Matrix (FIM) and Morimuras FIM, all RL algorithms with NG have followed the Kakades approach. In this paper, we describe a generalized Natural Gradient (gNG) by linearly interpolating the two FIMs and propose an efficient implementation for the gNG learning based on a theory of the estimating function, generalized Natural Actor-Critic (gNAC). The gNAC algorithm involves a near optimal auxiliary function to reduce the variance of the gNG estimates. Interestingly, the gNAC can be regarded as a natural extension of the current state-of-the-art NAC algorithm, as long as the interpolating parameter is appropriately selected. Numerical experiments showed that the proposed gNAC algorithm can estimate gNG efficiently and outperformed the NAC algorithm.

YNIMG Journal 2009 Journal Article

A hierarchical Bayesian method to resolve an inverse problem of MEG contaminated with eye movement artifacts

• Yusuke Fujiwara
• Okito Yamashita
• Dai Kawawaki
• Kenji Doya
• Mitsuo Kawato
• Keisuke Toyama
• Masa-aki Sato

The magnetic fields generated by eye movements are major artifacts in MEG measurements. We propose a hybrid hierarchical variational Bayesian method to remove eye movement artifacts from MEG data. Our method is an extension of the hierarchical variational Bayesian method for MEG source localization proposed by Sato et al. [Sato, M. , Yoshioka, T. , Kajihara, S. , Toyama, K. , Goda, N. , Doya, K. , and Kawato, M. , (2004). Hierarchical Bayesian estimation for MEG inverse problem. NeuroImage 23(3), 806–826]. First, we assumed a single dipole at each left and right eyeball as a source of eye artifacts. Second, we constructed an EOG forward model describing the relationship between eye dipoles and electric potentials, i. e. , EOG. Based on the Bayesian framework, the proposed method concurrently estimates eye and brain current sources from both MEG and EOG data. Thereby the brain current sources can be isolated from eye artifacts. The new method was tested in two ways. In the simulation experiments, the performance of eye artifact removal was evaluated from various aspects; locations of brain current sources, temporal correlation between eye and brain current sources, the level of MEG observation noise and so on. In real MEG experiments, we measured MEG and EOG data during smooth pursuit eye movements for a horizontally or circularly moving target. Our method successfully removed eye artifacts from the simulated and real MEG data with the estimation of brain current sources that were located in eye movement related areas. Our method should be widely applicable to MEG data obtained in tasks with non-negligible eye movements.

ICRA Conference 2005 Conference Paper

Learning Sensory Feedback to CPG with Policy Gradient for Biped Locomotion

• Takamitsu Matsubara
• Jun Morimoto
• Jun Nakanishi
• Masa-aki Sato
• Kenji Doya

This paper proposes a learning framework for a CPG-based biped locomotion controller using a policy gradient method. Our goal in this study is to develop an efficient learning algorithm by reducing the dimensionality of the state space used for learning. We demonstrate that an appropriate feedback controller in the CPG-based controller can be acquired using the proposed method within a few thousand trials by numerical simulations. Furthermore, we implement the learned controller on the physical biped robot to experimentally show that the learned controller successfully works in the real environment.

YNIMG Journal 2004 Journal Article

Hierarchical Bayesian estimation for MEG inverse problem

• Masa-aki Sato
• Taku Yoshioka
• Shigeki Kajihara
• Keisuke Toyama
• Naokazu Goda
• Kenji Doya
• Mitsuo Kawato

Source current estimation from MEG measurement is an ill-posed problem that requires prior assumptions about brain activity and an efficient estimation algorithm. In this article, we propose a new hierarchical Bayesian method introducing a hierarchical prior that can effectively incorporate both structural and functional MRI data. In our method, the variance of the source current at each source location is considered an unknown parameter and estimated from the observed MEG data and prior information by using the Variational Bayesian method. The fMRI information can be imposed as prior information on the variance distribution rather than the variance itself so that it gives a soft constraint on the variance. A spatial smoothness constraint, that the neural activity within a few millimeter radius tends to be similar due to the neural connections, can also be implemented as a hierarchical prior. The proposed method provides a unified theory to deal with the following three situations: (1) MEG with no other data, (2) MEG with structural MRI data on cortical surfaces, and (3) MEG with both structural MRI and fMRI data. We investigated the performance of our method and conventional linear inverse methods under these three conditions. Simulation results indicate that our method has better accuracy and spatial resolution than the conventional linear inverse methods under all three conditions. It is also shown that accuracy of our method improves as MRI and fMRI information becomes available. Simulation results demonstrate that our method appropriately resolves the inverse problem even if fMRI data convey inaccurate information, while the Wiener filter method is seriously deteriorated by inaccurate fMRI information.

IROS Conference 2004 Conference Paper

Multi-agent reinforcement learning: using macro actions to learn a mating task

• Stefan Elfwing
• Eiji Uchibe
• Kenji Doya
• Henrik I. Christensen

Standard reinforcement learning methods are inefficient and often inadequate for learning cooperative multi-agent tasks. For these kinds of tasks the behavior of one agent strongly depends on dynamic interaction with other agents, not only with the interaction with a static environment as in standard reinforcement learning. The success of the learning is therefore coupled to the agents' ability to predict the other agents behaviors. In this study we try to overcome this problem by adding a few simple macro actions, actions that are extended in time for more than one time step. The macro actions improve the learning by making search of the state space more effective and thereby making the behavior more predictable for the other agent. In this study we have considered a cooperative mating task, which is the first step towards our aim to perform embodied evolution, where the evolutionary selection process is an integrated part of the task. We show, in simulation and hardware, that in the case of learning without macro actions, the agents fail to learn a meaningful behavior. In contrast, for the learning with macro action the agents learn a good mating behavior in reasonable time, in both simulation and hardware.

NeurIPS Conference 2004 Conference Paper

Responding to Modalities with Different Latencies

• Fredrik Bissmarck
• Hiroyuki Nakahara
• Kenji Doya
• Okihide Hikosaka

Motor control depends on sensory feedback in multiple modalities with different latencies. In this paper we consider within the framework of re- inforcement learning how different sensory modalities can be combined and selected for real-time, optimal movement control. We propose an actor-critic architecture with multiple modules, whose output are com- bined using a softmax function. We tested our architecture in a simu- lation of a sequential reaching task. Reaching was initially guided by visual feedback with a long latency. Our learning scheme allowed the agent to utilize the somatosensory feedback with shorter latency when the hand is near the experienced trajectory. In simulations with different latencies for visual and somatosensory feedback, we found that the agent depended more on feedback with shorter latency.

NeurIPS Conference 2003 Conference Paper

Different Cortico-Basal Ganglia Loops Specialize in Reward Prediction at Different Time Scales

• Saori Tanaka
• Kenji Doya
• Go Okada
• Kazutaka Ueda
• Yasumasa Okamoto
• Shigeto Yamawaki

To understand the brain mechanisms involved in reward prediction on different time scales, we developed a Markov decision task that requires prediction of both immediate and future rewards, and ana- lyzed subjects’ brain activities using functional MRI. We estimated the time course of reward prediction and reward prediction error on different time scales from subjects' performance data, and used them as the explanatory variables for SPM analysis. We found topog- raphic maps of different time scales in medial frontal cortex and striatum. The result suggests that different cortico-basal ganglia loops are specialized for reward prediction on different time scales.

NeurIPS Conference 2003 Conference Paper

Estimating Internal Variables and Paramters of a Learning Agent by a Particle Filter

• Kazuyuki Samejima
• Kenji Doya
• Yasumasa Ueda
• Minoru Kimura

When we model a higher order functions, such as learning and memory, we face a difficulty of comparing neural activities with hidden variables that depend on the history of sensory and motor signals and the dynam- ics of the network. Here, we propose novel method for estimating hidden variables of a learning agent, such as connection weights from sequences of observable variables. Bayesian estimation is a method to estimate the posterior probability of hidden variables from observable data sequence using a dynamic model of hidden and observable variables. In this pa- per, we apply particle filter for estimating internal parameters and meta- parameters of a reinforcement learning model. We verified the effective- ness of the method using both artificial data and real animal behavioral data.

IROS Conference 2003 Conference Paper

Evolution of meta-parameters in reinforcement learning algorithm

• Anders Eriksson
• Genci Capi
• Kenji Doya

A crucial issue in reinforcement learning applications is how to set meta-parameters, such as the learning rate and "temperature" for exploration, to match the demands of the task and the environment. In this paper, we propose a method to adjust meta-parameters of reinforcement learning by real-number genetic algorithm. It was shown in simulations of foraging tasks that appropriate settings of meta-parameters, which are strongly dependent on each other, can be found by evolution. Furthermore, we verified in hardware experiments using cyber rodent (CR) robots that the meta-parameters evolved in simulation are helpful for learning in real hardware.

ICML Conference 2000 Conference Paper

Acquisition of Stand-up Behavior by a Real Robot using Hierarchical Reinforcement Learning

• Jun Morimoto
• Kenji Doya

NeurIPS Conference 2000 Conference Paper

Robust Reinforcement Learning

• Jun Morimoto
• Kenji Doya

This paper proposes a new reinforcement learning (RL) paradigm that explicitly takes into account input disturbance as well as mod(cid: 173) eling errors. The use of environmental models in RL is quite pop(cid: 173) ular for both off-line learning by simulations and for on-line ac(cid: 173) tion planning. However, the difference between the model and the real environment can lead to unpredictable, often unwanted results. Based on the theory of H oocontrol, we consider a differential game in which a 'disturbing' agent (disturber) tries to make the worst possible disturbance while a 'control' agent (actor) tries to make the best control input. The problem is formulated as finding a min(cid: 173) max solution of a value function that takes into account the norm of the output deviation and the norm of the disturbance. We derive on-line learning algorithms for estimating the value function and for calculating the worst disturbance and the best control in refer(cid: 173) ence to the value function. We tested the paradigm, which we call "Robust Reinforcement Learning (RRL), " in the task of inverted pendulum. In the linear domain, the policy and the value func(cid: 173) tion learned by the on-line algorithms coincided with those derived analytically by the linear H ootheory. For a fully nonlinear swing(cid: 173) up task, the control by RRL achieved robust performance against changes in the pendulum weight and friction while a standard RL control could not deal with such environmental changes.

IROS Conference 1998 Conference Paper

Reinforcement learning of dynamic motor sequence: learning to stand up

• Jun Morimoto
• Kenji Doya

We propose a learning method for implementing human-like sequential movements in robots. As an example of dynamic sequential movement, we consider the "stand-up" task for a two-joint, three-link robot. In contrast to the case of steady walking or standing, the desired trajectory for such a transient behavior is very difficult to derive. The goal of the task is to find a path that links a lying state to an upright state under the constraints of the system dynamics. The geometry of the robot is such that there is no static solution; the robot has to stand up dynamically utilizing the momentum of its body. We use reinforcement learning, in particular, a continuous time and state temporal difference (TD) learning method. For successful results, we use 1) an efficient method of value function approximation in a high-dimensional state space, and 2) a hierarchical architecture which divides a large state space into a few smaller pieces.

NeurIPS Conference 1996 Conference Paper

Efficient Nonlinear Control with Actor-Tutor Architecture

• Kenji Doya

A new reinforcement learning architecture for nonlinear control is proposed. A direct feedback controller, or the actor, is trained by a value-gradient based controller, or the tutor. This architecture enables both efficient use of the value function and simple computa(cid: 173) tion for real-time implementation. Good performance was verified in multi-dimensional nonlinear control tasks using Gaussian soft(cid: 173) max networks.

NeurIPS Conference 1995 Conference Paper

Dynamics of Attention as Near Saddle-Node Bifurcation Behavior

• Hiroyuki Nakahara
• Kenji Doya

In consideration of attention as a means for goal-directed behav(cid: 173) ior in non-stationary environments, we argue that the dynamics of attention should satisfy two opposing demands: long-term main(cid: 173) tenance and quick transition. These two characteristics are con(cid: 173) tradictory within the linear domain. We propose the near saddle(cid: 173) node bifurcation behavior of a sigmoidal unit with self-connection as a candidate of dynamical mechanism that satisfies both of these demands. We further show in simulations of the 'bug-eat-food' tasks that the near saddle-node bifurcation behavior of recurrent networks can emerge as a functional property for survival in non(cid: 173) stationary environments.

NeurIPS Conference 1995 Conference Paper

Temporal Difference Learning in Continuous Time and Space

• Kenji Doya

A continuous-time, continuous-state version of the temporal differ(cid: 173) ence (TD) algorithm is derived in order to facilitate the application of reinforcement learning to real-world control tasks and neurobi(cid: 173) ological modeling. An optimal nonlinear feedback control law was also derived using the derivatives of the value function. The per(cid: 173) formance of the algorithms was tested in a task of swinging up a pendulum with limited torque. Both the "critic" that specifies the paths to the upright position and the "actor" that works as a non(cid: 173) linear feedback controller were successfully implemented by radial basis function (RBF) networks.

NeurIPS Conference 1994 Conference Paper

A Novel Reinforcement Model of Birdsong Vocalization Learning

• Kenji Doya
• Terrence Sejnowski

Songbirds learn to imitate a tutor song through auditory and motor learn(cid: 173) ing. We have developed a theoretical framework for song learning that accounts for response properties of neurons that have been observed in many of the nuclei that are involved in song learning. Specifically, we suggest that the anteriorforebrain pathway, which is not needed for song production in the adult but is essential for song acquisition, provides synaptic perturbations and adaptive evaluations for syllable vocalization learning. A computer model based on reinforcement learning was con(cid: 173) structed that could replicate a real zebra finch song with 90% accuracy based on a spectrographic measure. The second generation of the bird(cid: 173) song model replicated the tutor song with 96% accuracy.

NeurIPS Conference 1993 Conference Paper

A Hodgkin-Huxley Type Neuron Model That Learns Slow Non-Spike Oscillation

• Kenji Doya
• Allen Selverston
• Peter Rowat

A gradient descent algorithm for parameter estimation which is similar to those used for continuous-time recurrent neural networks was derived for Hodgkin-Huxley type neuron models. Using mem(cid: 173) brane potential trajectories as targets, the parameters (maximal conductances, thresholds and slopes of activation curves, time con(cid: 173) stants) were successfully estimated. The algorithm was applied to modeling slow non-spike oscillation of an identified neuron in the lobster stomatogastric ganglion. A model with three ionic currents was trained with experimental data. It revealed a novel role of A-current for slow oscillation below -50 mY.

NeurIPS Conference 1992 Conference Paper

Mapping Between Neural and Physical Activities of the Lobster Gastric Mill

• Kenji Doya
• Mary Boyle
• Allen Selverston

A computer model of the musculoskeletal system of the lobster gastric mill was constructed in order to provide a behavioral in(cid: 173) terpretation of the rhythmic patterns obtained from isolated stom(cid: 173) atogastric ganglion. The model was based on Hill's muscle model and quasi-static approximation of the skeletal dynamics and could simulate the change of chewing patterns by the effect of neuromod(cid: 173) ulators. 1 THE STOMATOGASTRIC NERVOUS SYSTEM The crustacean stomatogastric ganglion (STG) is a circuit of 30 neurons that con(cid: 173) trols rhythmic movement of the foregut. It is one of the best elucidated neural circuits. All the neurons and the synaptic connections between them are identi(cid: 173) fied and the effects of neuromodulators on the oscillation patterns and neuronal characteristics have been extensively studied (Selverston and Moulins 1987, H arris(cid: 173) Warrick et al. 1992). However, STG's function as a controller of ingestive behavior is not fully understood in part because of our poor understanding of the controlled object: the musculoskeletal dynamics of the foregut. We constructed a mathemat(cid: 173) ical model of the gastric mill, three teeth in the stomach, in order to predict motor patterns from the neural oscillation patterns which are recorded from the isolated ganglion. The animal we used was the Californian spiny lobster (Panulirus interruptus), which

NeurIPS Conference 1991 Conference Paper

Adaptive Synchronization of Neural and Physical Oscillators

• Kenji Doya
• Shuji Yoshizawa

Animal locomotion patterns are controlled by recurrent neural networks called central pattern generators (CPGs). Although a CPG can oscillate autonomously, its rhythm and phase must be well coordinated with the state of the physical system using sensory inputs. In this paper we propose a learning algorithm for synchronizing neural and physical oscillators with specific phase relationships. Sensory input connections are modified by the correlation between cellular activities and input signals. Simulations show that the learning rule can be used for setting sensory feedback connections to a CPG as well as coupling connections between CPGs. 1 CENTRAL AND SENSORY MECHANISMS IN LOCOMOTION CONTROL Patterns of animal locomotion, such as walking, swimming, and fiying, are generated by recurrent neural networks that are located in segmental ganglia of invertebrates and spinal cords of vertebrates (Barnes and Gladden, 1985). These networks can produce basic rhythms of locomotion without sensory inputs and are called central pattern generators (CPGs). The physical systems of locomotion, such as legs, fins, and wings combined with physical environments, have their own oscillatory char(cid: 173) acteristics. Therefore, in order to realize efficient locomotion, the frequency and the phase of oscillation of a CPG must be well coordinated with the state of the physical system. For example, the bursting patterns of motoneurons that drive a leg muscle must be coordinated with the configuration of the leg, its contact with the ground, and the state of other legs.