Hirotaka Hachiya

ICML Conference 2013 Conference Paper

Squared-loss Mutual Information Regularization: A Novel Information-theoretic Approach to Semi-supervised Learning

• Gang Niu 0001
• Wittawat Jitkrittum
• Bo Dai 0001
• Hirotaka Hachiya
• Masashi Sugiyama

We propose squared-loss mutual information regularization (SMIR) for multi-class probabilistic classification, following the information maximization principle. SMIR is convex under mild conditions and thus improves the nonconvexity of mutual information regularization. It offers all of the following four abilities to semi-supervised algorithms: Analytical solution, out-of-sample/multi-class classification, and probabilistic output. Furthermore, novel generalization error bounds are derived. Experiments show SMIR compares favorably with state-of-the-art methods.

ICML Conference 2012 Conference Paper

Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting

• Ning Xie 0003
• Hirotaka Hachiya
• Masashi Sugiyama

NeurIPS Conference 2011 Conference Paper

Analysis and Improvement of Policy Gradient Estimation

• Tingting Zhao
• Hirotaka Hachiya
• Gang Niu
• Masashi Sugiyama

Policy gradient is a useful model-free reinforcement learning approach, but it tends to suffer from instability of gradient estimates. In this paper, we analyze and improve the stability of policy gradient methods. We first prove that the variance of gradient estimates in the PGPE(policy gradients with parameter-based exploration) method is smaller than that of the classical REINFORCE method under a mild assumption. We then derive the optimal baseline for PGPE, which contributes to further reducing the variance. We also theoretically show that PGPE with the optimal baseline is more preferable than REINFORCE with the optimal baseline in terms of the variance of gradient estimates. Finally, we demonstrate the usefulness of the improved PGPE method through experiments.

ICML Conference 2011 Conference Paper

On Information-Maximization Clustering: Tuning Parameter Selection and Analytic Solution

• Masashi Sugiyama
• Makoto Yamada
• Manabu Kimura
• Hirotaka Hachiya

NeurIPS Conference 2011 Conference Paper

Relative Density-Ratio Estimation for Robust Distribution Comparison

• Makoto Yamada
• Taiji Suzuki
• Takafumi Kanamori
• Hirotaka Hachiya
• Masashi Sugiyama

Divergence estimators based on direct approximation of density-ratios without going through separate approximation of numerator and denominator densities have been successfully applied to machine learning tasks that involve distribution comparison such as outlier detection, transfer learning, and two-sample homogeneity test. However, since density-ratio functions often possess high fluctuation, divergence estimation is still a challenging task in practice. In this paper, we propose to use relative divergences for distribution comparison, which involves approximation of relative density-ratios. Since relative density-ratios are always smoother than corresponding ordinary density-ratios, our proposed method is favorable in terms of the non-parametric convergence speed. Furthermore, we show that the proposed divergence estimator has asymptotic variance independent of the model complexity under a parametric setup, implying that the proposed estimator hardly overfits even with complex models. Through experiments, we demonstrate the usefulness of the proposed approach.

ICML Conference 2010 Conference Paper

Nonparametric Return Distribution Approximation for Reinforcement Learning

• Tetsuro Morimura
• Masashi Sugiyama
• Hisashi Kashima
• Hirotaka Hachiya
• Toshiyuki Tanaka 0003

UAI Conference 2010 Conference Paper

Parametric Return Density Estimation for Reinforcement Learning

• Tetsuro Morimura
• Masashi Sugiyama
• Hisashi Kashima
• Hirotaka Hachiya
• Toshiyuki Tanaka 0003

Most conventional Reinforcement Learning (RL) algorithms aim to optimize decisionmaking rules in terms of the expected returns. However, especially for risk management purposes, other risk-sensitive criteria such as the value-at-risk or the expected shortfall are sometimes preferred in real applications. Here, we describe a parametric method for estimating density of the returns, which allows us to handle various criteria in a unified manner. We first extend the Bellman equation for the conditional expected return to cover a conditional probability density of the returns. Then we derive an extension of the TD-learning algorithm for estimating the return densities in an unknown environment. As test instances, several parametric density estimation algorithms are presented for the Gaussian, Laplace, and skewed Laplace distributions. We show that these algorithms lead to risk-sensitive as well as robust RL paradigms through numerical experiments.

IJCAI Conference 2009 Conference Paper

• Takayuki Akiyama
• Hirotaka Hachiya
• Masashi Sugiyama

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. We demonstrate the usefulness of the proposed method, named active policy iteration (API), through simulations with a batting robot.

ICRA Conference 2009 Conference Paper

Least absolute policy iteration for robust value function approximation

• Masashi Sugiyama
• Hirotaka Hachiya
• Hisashi Kashima
• Tetsuro Morimura

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational efficiency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved efficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through simulated robot-control tasks.

AAAI Conference 2008 Conference Paper

Adaptive Importance Sampling with Automatic Model Selection in Value Function Approximation

• Hirotaka Hachiya
• Masashi Sugiyama

Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past, which is an essential problem for physically grounded AI as experiments are usually prohibitively expensive. A common approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-sampling policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations.

ICRA Conference 2007 Conference Paper

Value Function Approximation on Non-Linear Manifolds for Robot Motor Control

• Masashi Sugiyama
• Hirotaka Hachiya
• Christopher Towell
• Sethu Vijayakumar

The least squares approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in a simulated robot arm control and Khepera robot navigation.