Hiroshi Mamitsuka

UAI Conference 2025 Conference Paper

Multiple Wasserstein Gradient Descent Algorithm for Multi-Objective Distributional Optimization

• Dai Hai Nguyen
• Hiroshi Mamitsuka
• Atsuyoshi Nakamura

We address the optimization problem of simultaneously minimizing multiple objective functionals over a family of probability distributions. This type of Multi-Objective Distributional Optimization commonly arises in machine learning and statistics, with applications in areas such as multiple target sampling, multi-task learning, and multi-objective generative modeling. To solve this problem, we propose an iterative particle-based algorithm, which we call Muliple Wasserstein Gradient Descent (MWGraD), which constructs a flow of intermediate empirical distributions, each being represented by a set of particles, which gradually minimize the multiple objective functionals simultaneously. Specifically, MWGraD consists of two key steps at each iteration. First, it estimates the Wasserstein gradient for each objective functional based on the current particles. Then, it aggregates these gradients into a single Wasserstein gradient using dynamically adjusted weights and updates the particles accordingly. In addition, we provide theoretical analysis and present experimental results on both synthetic and real-world datasets, demonstrating the effectiveness of MWGraD.

IJCAI Conference 2024 Conference Paper

Learning Low-Rank Tensor Cores with Probabilistic ℓ0-Regularized Rank Selection for Model Compression

• Tianxiao Cao
• Lu Sun
• Canh Hao Nguyen
• Hiroshi Mamitsuka

Compressing deep neural networks is of great importance for real-world applications on resource-constrained devices. Tensor decomposition is one promising answer that retains the functionality and most of the expressive power of the original deep models by replacing the weights with their decomposed cores. Decomposition with optimal ranks can achieve a good compression-accuracy trade-off, but it is expensive to optimize due to its discrete and combinatorial nature. A common practice is to set all ranks equal and tune one hyperparameter, but it may significantly harm the flexibility and generalization. In this paper, we propose a novel automatic rank selection method for deep model compression that allows learning model weights and decomposition ranks simultaneously. We propose to penalize the ℓ0 (quasi-)norm of the slices of decomposed tensor cores during model training. To avoid combinatorial optimization, we develop a probabilistic formulation and apply an approximate Bernoulli gate to each of the slices of tensor cores, which can be implemented in an end-to-end and scalable framework via gradient descent. It enables the automatic rank selection to be incorporated with arbitrary tensor decompositions and neural network layers such as linear layers, convolutional layers, and embedding layers. Comprehensive experiments on various tasks, including image classification, text sentiment classification, and neural machine translation, demonstrate the superior effectiveness of the proposed method over baselines.

ECAI Conference 2023 Conference Paper

Multiplicative Sparse Tensor Factorization for Multi-View Multi-Task Learning

• Xinyi Wang
• Lu Sun 0001
• Canh Hao Nguyen
• Hiroshi Mamitsuka

Multi-View Multi-Task Learning (MVMTL) aims to make predictions on dual-heterogeneous data. Such data contains features from multiple views, and multiple tasks in the data are related with each other through common views. Existing MVMTL methods usually face two major challenges: 1) to save the predictive information from full-order interactions between views efficiently. 2) to learn a parsimonious and highly interpretable model such that the target is related to the features through a subset of interactions. To deal with the challenges, we propose a novel MVMTL method based on multiplicative sparse tensor factorization. For 1), we represent full-order interactions between views as a tensor, that enables to capture the complex correlations in dual-heterogeneous data by a concise model. For 2), we decompose the interaction tensor into a product of two components: one being shared with all tasks and the other being specific to individual tasks. Moreover, tensor factorization is applied to control the model complexity and learn a consensus latent representation shared by multiple tasks. Theoretical analysis reveals the equivalence between our method and a family of models with a joint but more general form of regularizers. Experiments on both synthetic and real-world datasets prove its effectiveness.

AAAI Conference 2020 Conference Paper

Efficiently Enumerating Substrings with Statistically Significant Frequencies of Locally Optimal Occurrences in Gigantic String

• Atsuyoshi Nakamura
• Ichigaku Takigawa
• Hiroshi Mamitsuka

We propose new frequent substring pattern mining which can enumerate all substrings with statistically significant frequencies of their locally optimal occurrences from a given single sequence. Our target application is genome sequences, around a half being said to be covered by interspersed and consecutive (tandem) repeats, and detecting these repeats is an important task in molecular life sciences. We evaluate the statistical significance of frequent substrings by using a string generation model with a memoryless stationary information source. We combine this idea with an existing algorithm, ESFLOO-0G. C (Nakamura et al. 2016), to enumerate all statistically significant substrings with locally optimal occurrences. We further develop a parallelized version of our algorithm. Experimental results using synthetic datasets showed the proposed algorithm achieved far higher F-measure in extracting substrings (with various lengths and frequencies) embedded in a randomly generated string with noise, than conventional algorithms. The large-scale experiment using the whole human genome sequence with 3, 095, 677, 412 bases (letters) showed that our parallel algorithm covers 75% of the whole positions analyzed, around 4% and 24% higher than the recent report and the current cutting-edge knowledge, implying a biologically unique finding.

NeurIPS Conference 2019 Conference Paper

AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification

• Ronghui You
• Zihan Zhang
• Ziye Wang
• Suyang Dai
• Hiroshi Mamitsuka
• Shanfeng Zhu

Extreme multi-label text classification (XMTC) is an important problem in the era of {\it big data}, for tagging a given text with the most relevant multiple labels from an extremely large-scale label set. XMTC can be found in many applications, such as item categorization, web page tagging, and news annotation. Traditionally most methods used bag-of-words (BOW) as inputs, ignoring word context as well as deep semantic information. Recent attempts to overcome the problems of BOW by deep learning still suffer from 1) failing to capture the important subtext for each label and 2) lack of scalability against the huge number of labels. We propose a new label tree-based deep learning model for XMTC, called AttentionXML, with two unique features: 1) a multi-label attention mechanism with raw text as input, which allows to capture the most relevant part of text to each label; and 2) a shallow and wide probabilistic label tree (PLT), which allows to handle millions of labels, especially for "tail labels". We empirically compared the performance of AttentionXML with those of eight state-of-the-art methods over six benchmark datasets, including Amazon-3M with around 3 million labels. AttentionXML outperformed all competing methods under all experimental settings. Experimental results also show that AttentionXML achieved the best performance against tail labels among label tree-based methods. The code and datasets are available at \url{http: //github. com/yourh/AttentionXML}.

IJCAI Conference 2019 Conference Paper

Fast and Robust Multi-View Multi-Task Learning via Group Sparsity

• Lu Sun
• Canh Hao Nguyen
• Hiroshi Mamitsuka

Multi-view multi-task learning has recently attracted more and more attention due to its dual-heterogeneity, i. e. ,each task has heterogeneous features from multiple views, and probably correlates with other tasks via common views. Existing methods usually suffer from three problems: 1) lack the ability to eliminate noisy features, 2) hold a strict assumption on view consistency and 3) ignore the possible existence of task-view outliers. To overcome these limitations, we propose a robust method with joint group-sparsity by decomposing feature parameters into a sum of two components, in which one saves relevant features (for Problem 1) and flexible view consistency (for Problem 2), while the other detects task-view outliers (for Problem 3). With a global convergence property, we develop a fast algorithm to solve the optimization problem in a linear time complexity w. r. t. the number of features and labeled samples. Extensive experiments on various synthetic and real-world datasets demonstrate its effectiveness.

IJCAI Conference 2019 Conference Paper

Multiplicative Sparse Feature Decomposition for Efficient Multi-View Multi-Task Learning

• Lu Sun
• Canh Hao Nguyen
• Hiroshi Mamitsuka

Multi-view multi-task learning refers to dealing with dual-heterogeneous data, where each sample has multi-view features, and multiple tasks are correlated via common views. Existing methods do not sufficiently address three key challenges: (a) saving task correlation efficiently, (b) building a sparse model and (c) learning view-wise weights. In this paper, we propose a new method to directly handle these challenges based on multiplicative sparse feature decomposition. For (a), the weight matrix is decomposed into two components via low-rank constraint matrix factorization, which saves task correlation by learning a reduced number of model parameters. For (b) and (c), the first component is further decomposed into two sub-components, to select topic-specific features and learn view-wise importance, respectively. Theoretical analysis reveals its equivalence with a general form of joint regularization, and motivates us to develop a fast optimization algorithm in a linear complexity w. r. t. the data size. Extensive experiments on both simulated and real-world datasets validate its efficiency.

NeurIPS Conference 2018 Conference Paper

Efficient Convex Completion of Coupled Tensors using Coupled Nuclear Norms

• Kishan Wimalawarne
• Hiroshi Mamitsuka

Coupled norms have emerged as a convex method to solve coupled tensor completion. A limitation with coupled norms is that they only induce low-rankness using the multilinear rank of coupled tensors. In this paper, we introduce a new set of coupled norms known as coupled nuclear norms by constraining the CP rank of coupled tensors. We propose new coupled completion models using the coupled nuclear norms as regularizers, which can be optimized using computationally efficient optimization methods. We derive excess risk bounds for proposed coupled completion models and show that proposed norms lead to better performance. Through simulation and real-data experiments, we demonstrate that proposed norms achieve better performance for coupled completion compared to existing coupled norms.

IJCAI Conference 2016 Conference Paper

A Robust Convex Formulation for Ensemble Clustering

• Junning Gao
• Makoto Yamada
• Samuel Kaski
• Hiroshi Mamitsuka
• Shanfeng Zhu

We formulate ensemble clustering as a regularization problem over nuclear norm and cluster-wise group norm, and present an efficient optimization algorithm, which we call Robust Convex Ensemble Clustering (RCEC). A key feature of RCEC allows to remove anomalous cluster assignments obtained from component clustering methods by using the group-norm regularization. Moreover, the proposed method is convex and can find the globally optimal solution. We first showed that using synthetic data experiments, RCEC could learn stable cluster assignments from the input matrix including anomalous clusters. We then showed that RCEC outperformed state-of-the-art ensemble clustering methods by using real-world data sets.

IJCAI Conference 2015 Conference Paper

Instance-Wise Weighted Nonnegative Matrix Factorization for Aggregating Partitions with Locally Reliable Clusters

• Xiaodong Zheng
• Shanfeng Zhu
• Junning Gao
• Hiroshi Mamitsuka

We address an ensemble clustering problem, where reliable clusters are locally embedded in given multiple partitions. We propose a new nonnegative matrix factorization (NMF)-based method, in which locally reliable clusters are explicitly considered by using instance-wise weights over clusters. Our method factorizes the input cluster assignment matrix into two matrices H and W, which are optimized by iteratively 1) updating H and W while keeping the weight matrix constant and 2) updating the weight matrix while keeping H and W constant, alternatively. The weights in the second step were updated by solving a convex problem, which makes our algorithm significantly faster than existing NMF-based ensemble clustering methods. We empirically proved that our method outperformed a lot of cutting-edge ensemble clustering methods by using a variety of datasets.

NeurIPS Conference 2013 Conference Paper

Manifold-based Similarity Adaptation for Label Propagation

• Masayuki Karasuyama
• Hiroshi Mamitsuka

Label propagation is one of the state-of-the-art methods for semi-supervised learning, which estimates labels by propagating label information through a graph. Label propagation assumes that data points (nodes) connected in a graph should have similar labels. Consequently, the label estimation heavily depends on edge weights in a graph which represent similarity of each node pair. We propose a method for a graph to capture the manifold structure of input features using edge weights parameterized by a similarity function. In this approach, edge weights represent both similarity and local reconstruction weight simultaneously, both being reasonable for label propagation. For further justification, we provide analytical considerations including an interpretation as a cross-validation of a propagation model in the feature space, and an error analysis based on a low dimensional manifold model. Experimental results demonstrated the effectiveness of our approach both in synthetic and real datasets.

AIIM Journal 2005 Journal Article

Computational intelligence in solving bioinformatics problems

• Krzysztof J. Cios
• Hiroshi Mamitsuka
• Tomomasa Nagashima
• Ryszard Tadeusiewicz

AIIM Journal 2005 Journal Article

Finding the biologically optimal alignment of multiple sequences

• Hiroshi Mamitsuka

Objective: Deterministic annealing, which is derived from statistical physics, is a method for obtaining the global optimum in parameter space. During the annealing process, starting from high temperatures which are then lowered, deterministic annealing deterministically find the (global) optimum at each temperature. Thus, deterministic annealing is expected to be more computationally efficient than stochastic sampling strategies to obtain the global optimum. We propose to apply the deterministic annealing technique to the problem of efficiently finding the biologically optimal alignment of multiple sequences. Methods and material: We take a strategy based on probabilistic models for aligning multiple sequences. That is, we train a probabilistic model using given training sequences and obtain their alignment by parsing, i. e. searching for the most likely parse of each sequence and gaps using the trained parameters of the model. In this scenario, we propose a new stochastic model, which is simple enough to be suited to multiple sequence alignment and, unlike existing stochastic models, say a profile hidden Markov model (HMM), allows us to use similarity scores between symbols (or a symbol and a gap). We further present a learning algorithm for our simple model by combining deterministic annealing with an expectation–maximization (EM) algorithm. We emphasize that our approach is time-efficient, even if the training is done through an annealing process. Results: In our experiments, we used actual protein sequences whose three-dimensional (3D) structures are determined and which are all aligned based on their 3D structures. We compared the results obtained by our approach with those by other existing approaches. Experimental results clearly showed that our approach gave the best performance, in terms of the similarity to the structurally determined alignment, among the approaches tested. Experimental results further indicated that our approach was ten times more efficient in terms of actual computation time than a competing method.

ICML Conference 2003 Conference Paper

Hierarchical Latent Knowledge Analysis for Co-occurrence Data

• Hiroshi Mamitsuka

ICML Conference 2000 Conference Paper

Efficient Mining from Large Databases by Query Learning

• Hiroshi Mamitsuka
• Naoki Abe

ICML Conference 1998 Conference Paper

Query Learning Strategies Using Boosting and Bagging

• Naoki Abe
• Hiroshi Mamitsuka

ICML Conference 1994 Conference Paper

A New Method for Predicting Protein Secondary Structures Based on Stochastic Tree Grammars

• Naoki Abe
• Hiroshi Mamitsuka