Author name cluster

Ryutaro Ichise

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

5 papers

2 author rows

ECAI Conference 2025 Conference Paper

Improving Zero-Shot Generalization in Reinforcement Learning Through Abstract Representations

Tsai Shuo Kuo
Ryutaro Ichise

Reinforcement learning algorithms suffer from zero-shot generalization, where a learned policy is tested in new, out-of-distribution environments. One of the key reasons is the overfitting of the representation learning model. Previous studies have used data augmentation or contrastive learning to address this overfitting. Yet these methods often rely on task-specific augmentations or hand-picked negative samples, which are not robust to various tasks. We hypothesize that learning more abstract representations by ignoring pixel-level details allows the representation learning model to generalize better. Moreover, when an agent encounters a scene similar to one seen in the training environment but with slight variations in the background, it will ignore the differences and treat the scene as invariant, making the correct decision. Based on this idea, we introduce Dream with Abstractions (Dr. Abs), an algorithm that requires no custom augmentations or negative samples. First, we train a representation network with a joint embedding predictive architecture, where it learns an abstract representation. Next, we use the learned representation to regularize the latent states inside the reinforcement-learning agent. By ignoring pixel-level details, our method outperforms the baseline algorithm by approximately 22% on the widely used zero-shot generalization setup, thereby enhancing its generalization ability. Furthermore, Dr. Abs can be used in conjunction with other RL methods and can work in tandem with techniques such as data augmentation.

Details

AAMAS Conference 2022 Conference Paper

Goal-Driven Active Learning

Nicolas Bougie
Ryutaro Ichise

Despite recent breakthroughs for learning a rich set of behaviors in simulated tasks, reinforcement learning agents are not yet in widespread use in the real world where rewards are naturally sparse. In fact, efficient exploration remains a key challenge in sparse-reward tasks as it requires quickly finding informative and task-relevant experiences. While cloning behaviors provided by an expert is a promising approach to the exploration problem, learning from a fixed set of demonstrations may be impracticable due to lack of state coverage or distribution mismatch - when the learner’s goal deviates from the demonstrated behaviors. Moreover, we aim to obtain a policy that can accomplish a variety of goals guided by the same set of demonstrations (i. e. without additional human effort). We present a goal-conditioned method that leverages very small sets of goal-driven demonstrations to significantly accelerate learning. Crucially, we present the concept of active goal-driven demonstrations to query the demonstrator only in hard-to-learn and uncertain regions of the state space. We evaluate our framework on a set of robot control tasks. Our method outperforms prior imitation learning approaches in most of the tasks in terms of data efficiency and scores while reducing the amount of human effort.

PDF

JAAMAS Journal 2021 Journal Article

Goal-driven active learning

Nicolas Bougie
Ryutaro Ichise

Abstract Deep reinforcement learning methods have achieved significant successes in complex decision-making problems. In fact, they traditionally rely on well-designed extrinsic rewards, which limits their applicability to many real-world tasks where rewards are naturally sparse. While cloning behaviors provided by an expert is a promising approach to the exploration problem, learning from a fixed set of demonstrations may be impracticable due to lack of state coverage or distribution mismatch—when the learner’s goal deviates from the demonstrated behaviors. Besides, we are interested in learning how to reach a wide range of goals from the same set of demonstrations. In this work we propose a novel goal-conditioned method that leverages very small sets of goal-driven demonstrations to massively accelerate the learning process. Crucially, we introduce the concept of active goal-driven demonstrations to query the demonstrator only in hard-to-learn and uncertain regions of the state space. We further present a strategy for prioritizing sampling of goals where the disagreement between the expert and the policy is maximized. We evaluate our method on a variety of benchmark environments from the Mujoco domain. Experimental results show that our method outperforms prior imitation learning approaches in most of the tasks in terms of exploration efficiency and average scores.

Details DOI

IJCAI Conference 2020 Conference Paper

Towards High-Level Intrinsic Exploration in Reinforcement Learning

Nicolas Bougie
Ryutaro Ichise

Deep reinforcement learning (DRL) methods traditionally struggle with tasks where environment rewards are sparse or delayed, which entails that exploration remains one of the key challenges of DRL. Instead of solely relying on extrinsic rewards, many state-of-the-art methods use intrinsic curiosity as exploration signal. While they hold promise of better local exploration, discovering global exploration strategies is beyond the reach of current methods. We propose a novel end-to-end intrinsic reward formulation that introduces high-level exploration in reinforcement learning. Our curiosity signal is driven by a fast reward that deals with local exploration and a slow reward that incentivizes long-time horizon exploration strategies. We formulate curiosity as the error in an agent’s ability to reconstruct the observations given their contexts. Experimental results show that this high-level exploration enables our agents to outperform prior work in several Atari games.

PDF Details DOI

AAAI Conference 2018 Conference Paper

TorusE: Knowledge Graph Embedding on a Lie Group

Takuma Ebisu
Ryutaro Ichise

Knowledge graphs are useful for many artiﬁcial intelligence (AI) tasks. However, knowledge graphs often have missing facts. To populate the graphs, knowledge graph embedding models have been developed. Knowledge graph embedding models map entities and relations in a knowledge graph to a vector space and predict unknown triples by scoring candidate triples. TransE is the ﬁrst translation-based method and it is well known because of its simplicity and eﬃciency for knowledge graph completion. It employs the principle that the diﬀerences between entity embeddings represent their relations. The principle seems very simple, but it can eﬀectively capture the rules of a knowledge graph. However, TransE has a problem with its regularization. TransE forces entity embeddings to be on a sphere in the embedding vector space. This regularization warps the embeddings and makes it dif- ﬁcult for them to fulﬁll the abovementioned principle. The regularization also aﬀects adversely the accuracies of the link predictions. On the other hand, regularization is important because entity embeddings diverge by negative sampling without it. This paper proposes a novel embedding model, TorusE, to solve the regularization problem. The principle of TransE can be deﬁned on any Lie group. A torus, which is one of the compact Lie groups, can be chosen for the embedding space to avoid regularization. To the best of our knowledge, TorusE is the ﬁrst model that embeds objects on other than a real or complex vector space, and this paper is the ﬁrst to formally discuss the problem of regularization of TransE. Our approach outperforms other state-of-the-art approaches such as TransE, DistMult and ComplEx on a standard link prediction task. We show that TorusE is scalable to large-size knowledge graphs and is faster than the original TransE.

PDF Details