Arrow Research search

Author name cluster

Yichen Zhu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

20 papers
2 author rows

Possible papers

20

AAAI Conference 2025 Conference Paper

A Comprehensive Overhaul of Multimodal Assistant with Small Language Models

  • Minjie Zhu
  • Yichen Zhu
  • Ning Liu
  • Xin Liu
  • Zhiyuan Xu
  • Chaomin Shen
  • Yaxin Peng

Multimodal Large Language Models (MLLMs) have showcased impressive skills in tasks related to visual understanding and reasoning. Yet, their widespread application faces obstacles due to the high computational demands during both the training and inference phases, restricting their use to a limited audience within the research and user communities. In this paper, we investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha, which is designed to create synergy among various aspects: visual representation, language models, and optimization strategies. We show that without increasing the volume of training data, our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks. Through detailed discussion, we provide insights and guidelines for developing strong MSLMs that rival the capabilities of MLLMs.

PDF Details DOI

IJCAI Conference 2025 Conference Paper

Active Multimodal Distillation for Few-shot Action Recognition

  • Weijia Feng
  • Yichen Zhu
  • Ruojia Zhang
  • Chenyang Wang
  • Fei Ma
  • Xiaobao Wang
  • Xiaobai Li

Owing to its rapid progress and broad application prospects, few-shot action recognition has attracted considerable interest. However, current methods are predominantly based on limited single-modal data, which does not fully exploit the potential of multimodal information. This paper presents a novel framework that actively identifies reliable modalities for each sample using task-specific contextual cues, thus significantly improving recognition performance. Our framework integrates an Active Sample Inference (ASI) module, which utilizes active inference to predict reliable modalities based on posterior distributions and subsequently organizes them accordingly. Unlike reinforcement learning, active inference replaces rewards with evidence-based preferences, making more stable predictions. Additionally, we introduce an active mutual distillation module that enhances the representation learning of less reliable modalities by transferring knowledge from more reliable ones. Adaptive multimodal inference is employed during the meta-test to assign higher weights to reliable modalities. Extensive experiments across multiple benchmarks demonstrate that our method significantly outperforms existing approaches.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

ChatVLA-2: Vision-Language-Action Model with Open-World Reasoning

  • Zhongyi Zhou
  • Yichen Zhu
  • Xiaoyu Liu
  • Zhibin Tang
  • Junjie Wen
  • Yaxin Peng
  • Chaomin Shen
  • Yi Xu

Vision-language-action (VLA) models have emerged as the next generation of models in robotics. However, despite leveraging powerful pre-trained Vision-Language Models (VLMs), existing end-to-end VLA systems often lose key capabilities during fine-tuning as the model adapts to specific robotic tasks. We argue that a generalizable VLA model should retain and expand upon the VLM's core competencies: 1) **Open-world reasoning** - the VLA should inherit the knowledge from VLM, i. e. , recognize anything that the VLM can recognize, capable of solving math problems, possessing visual-spatial intelligence, 2) **Reasoning following** – effectively translating the open-world reasoning into actionable steps for the robot. In this work, we introduce **ChatVLA-2**, a novel mixture-of-expert VLA model coupled with a specialized three-stage training pipeline designed to preserve the VLM’s original strengths while enabling actionable reasoning. To validate our approach, we design a math-matching task wherein a robot interprets math problems written on a whiteboard and picks corresponding number cards from a table to solve equations. Remarkably, our method exhibits exceptional mathematical reasoning and OCR capabilities, despite these abilities not being explicitly trained within the VLA. Furthermore, we demonstrate that the VLA possesses strong spatial reasoning skills, enabling it to interpret novel directional instructions involving previously unseen objects. Overall, our method showcases reasoning and comprehension abilities that significantly surpass state-of-the-art imitation learning methods such as OpenVLA, DexVLA, and $\pi_0$. This work represents a substantial advancement toward developing truly generalizable robotic foundation models endowed with robust reasoning capacities.

PDF Details

NeurIPS Conference 2025 Conference Paper

Less is More: an Attention-free Sequence Prediction Modeling for Offline Embodied Learning

  • Wei Huang
  • Jianshu Zhang
  • Leiyu Wang
  • Heyue Li
  • Luoyi Fan
  • Yichen Zhu
  • Nanyang Ye
  • Qinying Gu

Offline reinforcement learning (offline RL) is increasingly approached as a sequence modeling task, with methods leveraging advanced architectures like Transformers to capture trajectory dependencies. Despite significant progress, the mechanisms underlying their effectiveness and limitations remain insufficiently understood. We conduct a thorough analysis on the representative Decision Transformer (DT) model using an entropy analysis and identify the inconsistencies in state-action-reward ($\langle s, a, R \rangle$) distributions causing attention ``dispersal". To address this, we propose a hierarchical framework that decomposes sequence modeling into intra-step relational modeling—handled by a Token Merger that fuses each $\langle s, a, R \rangle$ triplet—and inter-step modeling—handled by a Token Mixer across timesteps. We investigate several Token Merger designs and validate their effectiveness across various offline RL methods. Furthermore, our theoretical analysis and experimental results suggest that while Token Mixers are important, lightweight architecture can also achieve even better performance to more complex ones. We therefore propose a parameter-free Average Pooling Token Mixer, which, combined with a convolutional Token Merger, forms our final model, Decision HiFormer (DHi). DHi achieves a \textbf{73. 6\%} improvement in inference speed and an \textbf{9. 3\%} gain in policy performance on the D4RL benchmark compared to DT. DHi also generalizes well to real-world robotic manipulation tasks, offering both practical benefits and insights into sequence-based policy design for offline RL. Code and models are public at \href{https: //wei-nijuan. github. io/DecisionHiFormer/}{project page}.

PDF Details

NeurIPS Conference 2025 Conference Paper

SPMDM: Enhancing Masked Diffusion Models through Simplifying Sampling Path

  • Yichen Zhu
  • Weiyu Chen
  • James Kwok
  • Zhou Zhao

Autoregressive models (ARMs) show strong capabilities in many domains but face challenges with planning and complex reasoning due to their sequential generation. Masked diffusion models (MDMs) address these issues by enabling controllable, any-order, and parallel generation but encounter training difficulties as token prediction complexity varies with unmasked token positions. This work identifies two key characteristics of efficient MDM sampling paths: prioritizing tokens near unmasked ones and generating subsequence earlier in reasoning. We propose the Simple Path Masked Diffusion Model (SPMDM), which partitions sequences into fixed-length, non-overlapping subsequences and applies varying noise scales to learn token-level and cross-subsequence dependencies. Experiments on synthetic data and tasks like Countdown and Sudoku show SPMDM captures structural rules effectively, significantly outperforming existing MDMs and ARMs, with competitive results on broader reasoning benchmarks.

PDF Details

JBHI Journal 2025 Journal Article

Utilizing Artificial Intelligence (AI) and Image Recognition Technologies to achieve Potential Wireless Sensing of Surgical Resection Boundaries for Benign Prostatic Hyperplasia (BPH)

  • Meishan Zhao
  • Jingcheng Lv
  • Xuanhao Li
  • Tianming Cheng
  • Fangzhou Zhao
  • Liangshuo Zhang
  • Yichen Zhu
  • Mingjun Shi

Background and Objective: Precise intraoperative sensing is critical for optimizing surgical outcomes and patient safety. This study proposes an AI driven model, leveraging the nnU-Net architecture, to achieve real-time identification of surgical resection boundaries during transurethral resection of the prostate (TURP). While currently relying on image-based recognition, this work lays the foundation for integrating wireless sensing technologies, potentially transforming non-contact surgical guidance and biomedical applications. Methods: In this research, we presented an nnU Net model capable of automatically predicting benign prostatic hyperplasia tissues, specifically encompassing the prostatic capsule, verumontanum, and bladder neck. The model was constructed using a contracting path and an expanding path, with Leaky ReLUs employed to fine-tune the learning rate. We conducted separate evaluations to assess the AI aided recognition performance for the prostatic capsule, verumontanum, and bladder neck. Results: This study employed a 5-fold cross-validation approach to process each binary dataset. Specifically, for the prostate surgery capsule group, 228 images comprised the training set, while 58 images constituted the validation set. In the verumontanum group, 210 images were designated for training and 53 for validation. Similarly, in the bladder neck group, 236 samples were allocated to the training set and 59 to the validation set. The nnU-Net model was then trained and validated using these datasets, with its performance being assessed through the use of Dice coefficient, mean Intersection over Union (mIoU), mean Accuracy (mAcc), and overall Accuracy (aAcc) metrics. Prostate surgical capsule: mDice = 0. 67, mIoU = 0. 579, mAcc = 0. 7, aAcc = 0. 926; Verumontanum: mDice = 0. 837, mIoU = 0. 742, mAcc = 0. 835, aAcc = 0. 943; Bladder neck: mDice = 0. 76, mIoU = 0. 667, mAcc = 0. 748, aAcc = 0. 967These results demonstrate the model's performance in predicting and delineating these surgical boundaries. Conclusons Our findings highlight the potential of AI in real-time human sensing and pave the way for future integration with wireless technologies, enabling non-contact detection and precision healthcare solutions. This approach aligns with emerging trends in personalized biomedical sensing and wireless healthcare technologies.

Details DOI

NeurIPS Conference 2024 Conference Paper

Action Imitation in Common Action Space for Customized Action Image Synthesis

  • Wang Lin
  • Jingyuan Chen
  • Jiaxin Shi
  • Zirun Guo
  • Yichen Zhu
  • Zehan Wang
  • Tao Jin
  • Zhou Zhao

We propose a novel method, \textbf{TwinAct}, to tackle the challenge of decoupling actions and actors in order to customize the text-guided diffusion models (TGDMs) for few-shot action image generation. TwinAct addresses the limitations of existing methods that struggle to decouple actions from other semantics (e. g. , the actor's appearance) due to the lack of an effective inductive bias with few exemplar images. Our approach introduces a common action space, which is a textual embedding space focused solely on actions, enabling precise customization without actor-related details. Specifically, TwinAct involves three key steps: 1) Building common action space based on a set of representative action phrases; 2) Imitating the customized action within the action space; and 3) Generating highly adaptable customized action images in diverse contexts with action similarity loss. To comprehensively evaluate TwinAct, we construct a novel benchmark, which provides sample images with various forms of actions. Extensive experiments demonstrate TwinAct's superiority in generating accurate, context-independent customized actions while maintaining the identity consistency of different subjects, including animals, humans, and even customized actors.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Any2Policy: Learning Visuomotor Policy with Any-Modality

  • Yichen Zhu
  • Zhicai Ou
  • Feifei Feng
  • Jian Tang

Humans can communicate and observe media with different modalities, such as texts, sounds, and images. For robots to be more generalizable embodied agents, they should be capable of following instructions and perceiving the world with adaptation to diverse modalities. Current robotic learning methodologies often focus on single-modal task specification and observation, thereby limiting their ability to process rich multi-modal information. Addressing this limitation, we present an end-to-end general-purpose multi-modal system named Any-to-Policy Embodied Agents. This system empowers robots to handle tasks using various modalities, whether in combinations like text-image, audio-image, text-point cloud, or in isolation. Our innovative approach involves training a versatile modality network that adapts to various inputs and connects with policy networks for effective control. Because of the lack of existing multi-modal robotics datasets for evaluation, we assembled a comprehensive real-world dataset encompassing 30 robotic tasks. Each task in this dataset is richly annotated across multiple modalities, providing a robust foundation for assessment. We conducted extensive validation of our proposed unified modality embodied agent using several simulation benchmarks, including Franka Kitchen, Meta-World, and Maniskill2, as well as in our real-world settings. Our experiments showcase the promising capability of building embodied agents that can adapt to diverse multi-modal in a unified framework.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

EDT: An Efficient Diffusion Transformer Framework Inspired by Human-like Sketching

  • Xinwang Chen
  • Ning Liu
  • Yichen Zhu
  • Feifei Feng
  • Jian Tang

Transformer-based Diffusion Probabilistic Models (DPMs) have shown more potential than CNN-based DPMs, yet their extensive computational requirements hinder widespread practical applications. To reduce the computation budget of transformer-based DPMs, this work proposes the Efficient Diffusion Transformer (EDT) framework. This framework includes a lightweight-design diffusion model architecture, and a training-free Attention Modulation Matrix and its alternation arrangement in EDT inspired by human-like sketching. Additionally, we propose a token relation-enhanced masking training strategy tailored explicitly for EDT to augment its token relation learning capability. Our extensive experiments demonstrate the efficacy of EDT. The EDT framework reduces training and inference costs and surpasses existing transformer-based diffusion models in image synthesis performance, thereby achieving a significant overall enhancement. With lower FID, EDT-S, EDT-B, and EDT-XL attained speed-ups of 3. 93x, 2. 84x, and 1. 92x respectively in the training phase, and 2. 29x, 2. 29x, and 2. 22x respectively in inference, compared to the corresponding sizes of MDTv2. Our code is available at https: //github. com/xinwangChen/EDT.

PDF Details DOI

AAAI Conference 2024 Conference Paper

EPSD: Early Pruning with Self-Distillation for Efficient Model Compression

  • Dong Chen
  • Ning Liu
  • Yichen Zhu
  • Zhengping Che
  • Rui Ma
  • Fachao Zhang
  • Xiaofeng Mou
  • Yi Chang

Neural network compression techniques, such as knowledge distillation (KD) and network pruning, have received increasing attention. Recent work `Prune, then Distill' reveals that a pruned student-friendly teacher network can benefit the performance of KD. However, the conventional teacher-student pipeline, which entails cumbersome pre-training of the teacher and complicated compression steps, makes pruning with KD less efficient. In addition to compressing models, recent compression techniques also emphasize the aspect of efficiency. Early pruning demands significantly less computational cost in comparison to the conventional pruning methods as it does not require a large pre-trained model. Likewise, a special case of KD, known as self-distillation (SD), is more efficient since it requires no pre-training or student-teacher pair selection. This inspires us to collaborate early pruning with SD for efficient model compression. In this work, we propose the framework named Early Pruning with Self-Distillation (EPSD), which identifies and preserves distillable weights in early pruning for a given SD task. EPSD efficiently combines early pruning and self-distillation in a two-step process, maintaining the pruned network's trainability for compression. Instead of a simple combination of pruning and SD, EPSD enables the pruned network to favor SD by keeping more distillable weights before training to ensure better distillation of the pruned network. We demonstrated that EPSD improves the training of pruned networks, supported by visual and quantitative analyses. Our evaluation covered diverse benchmarks (CIFAR-10/100, Tiny-ImageNet, full ImageNet, CUB-200-2011, and Pascal VOC), with EPSD outperforming advanced pruning and SD techniques.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective

  • Wanying Wang
  • Yichen Zhu
  • Yirui Zhou
  • Chaomin Shen
  • Jian Tang
  • Zhiyuan Xu
  • Yaxin Peng
  • Yangchun Zhang

Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with stochastic policy (ST-GAIL). We begin with the observation that the training can be highly unstable for DE-GAIL at the beginning of the training phase and end up divergence. Conversely, the ST-GAIL training trajectory remains consistent, reliably converging. To shed light on these disparities, we provide an explanation from a theoretical perspective. By establishing a probabilistic lower bound for GAIL, we demonstrate that gradient explosion is an inevitable outcome for DE-GAIL due to occasionally large expert-imitator policy disparity, whereas ST-GAIL does not have the issue with it. To substantiate our assertion, we illustrate how modifications in the reward function can mitigate the gradient explosion challenge. Finally, we propose CREDO, a simple yet effective strategy that clips the reward function during the training phase, allowing the GAIL to enjoy high data efficiency and stable trainability.

PDF Details DOI

ICML Conference 2024 Conference Paper

Non-confusing Generation of Customized Concepts in Diffusion Models

  • Wang Lin
  • Jingyuan Chen
  • Jiaxin Shi
  • Yichen Zhu
  • Chen Liang
  • Junzhong Miao
  • Tao Jin 0004
  • Zhou Zhao 0001

We tackle the common challenge of inter-concept visual confusion in compositional concept generation using text-guided diffusion models (TGDMs). It becomes even more pronounced in the generation of customized concepts, due to the scarcity of user-provided concept visual examples. By revisiting the two major stages leading to the success of TGDMs—1) contrastive image-language pre-training (CLIP) for text encoder that encodes visual semantics, and 2) training TGDM that decodes the textual embeddings into pixels—we point that existing customized generation methods only focus on fine-tuning the second stage while overlooking the first one. To this end, we propose a simple yet effective solution called CLIF: contrastive image-language fine-tuning. Specifically, given a few samples of customized concepts, we obtain non-confusing textual embeddings of a concept by fine-tuning CLIP via contrasting a concept and the over-segmented visual regions of other concepts. Experimental results demonstrate the effectiveness of CLIF in preventing the confusion of multi-customized concept generation. Project page: https: //clif-official. github. io/clif.

Details

NeurIPS Conference 2024 Conference Paper

RAGraph: A General Retrieval-Augmented Graph Learning Framework

  • Xinke Jiang
  • Rihong Qiu
  • Yongxin Xu
  • Wentao Zhang
  • Yichen Zhu
  • Ruizhe Zhang
  • Yuchen Fang
  • Xu Chu

Graph Neural Networks (GNNs) have become essential in interpreting relational data across various domains, yet, they often struggle to generalize to unseen graph data that differs markedly from training instances. In this paper, we introduce a novel framework called General Retrieval-Augmented Graph Learning (RAGraph), which brings external graph data into the general graph foundation model to improve model generalization on unseen scenarios. On the top of our framework is a toy graph vector library that we established, which captures key attributes, such as features and task-specific label information. During inference, the RAGraph adeptly retrieves similar toy graphs based on key similarities in downstream tasks, integrating the retrieved data to enrich the learning context via the message-passing prompting mechanism. Our extensive experimental evaluations demonstrate that RAGraph significantly outperforms state-of-the-art graph learning methods in multiple tasks such as node classification, link prediction, and graph classification across both dynamic and static datasets. Furthermore, extensive testing confirms that RAGraph consistently maintains high performance without the need for task-specific fine-tuning, highlighting its adaptability, robustness, and broad applicability.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Safety of Multimodal Large Language Models on Images and Text

  • Xin Liu
  • Yichen Zhu
  • Yunshi Lan
  • Chao Yang
  • Yu Qiao

Attracted by the impressive power of Multimodal Large Language Models (MLLMs), the public is increasingly utilizing them to improve the efficiency of daily work. Nonetheless, the vulnerabilities of MLLMs to unsafe instructions bring huge safety risks when these models are deployed in real-world scenarios. In this paper, we systematically survey current efforts on the evaluation, attack, and defense of MLLMs' safety on images and text. We begin with introducing the overview of MLLMs on images and text and understanding of safety, which helps researchers know the detailed scope of our survey. Then, we review the evaluation datasets and metrics for measuring the safety of MLLMs. Next, we comprehensively present attack and defense techniques related to MLLMs' safety. Finally, we analyze several unsolved issues and discuss promising research directions. The relevant papers are collected at "https: //github. com/isXinLiu/Awesome-MLLM-Safety".

PDF Details DOI

IJCAI Conference 2023 Conference Paper

Prediction with Incomplete Data under Agnostic Mask Distribution Shift

  • Yichen Zhu
  • Jian Yuan
  • Bo Jiang
  • Tao Lin
  • Haiming Jin
  • Xinbing Wang
  • Chenghu Zhou

Data with missing values is ubiquitous in many applications. Recent years have witnessed increasing attention on prediction with only incomplete data consisting of observed features and a mask that indicates the missing pattern. Existing methods assume that the training and testing distributions are the same, which may be violated in real-world scenarios. In this paper, we consider prediction with incomplete data in the presence of distribution shift. We focus on the case where the underlying joint distribution of complete features and label is invariant, but the missing pattern, i. e. , mask distribution may shift agnostically between training and testing. To achieve generalization, we leverage the observation that for each mask, there is an invariant optimal predictor. To avoid the exponential explosion when learning them separately, we approximate the optimal predictors jointly using a double parameterization technique. This has the undesirable side effect of allowing the learned predictors to rely on the intra-mask correlation and that between features and mask. We perform decorrelation to minimize this effect. Combining the techniques above, we propose a novel prediction method called StableMiss. Extensive experiments on both synthetic and real-world datasets show that StableMiss is robust and outperforms state-of-the-art methods under agnostic mask distribution shift.

PDF Details DOI

IROS Conference 2023 Conference Paper

Revisiting Event-Based Video Frame Interpolation

  • Jiaben Chen
  • Yichen Zhu
  • Dongze Lian
  • Jiaqi Yang
  • Yifu Wang
  • Renrui Zhang
  • Xinhang Liu
  • Shenhan Qian

Dynamic vision sensors or event cameras provide rich complementary information for video frame interpolation. Existing state-of-the-art methods follow the paradigm of combining both synthesis-based and warping networks. However, few of those methods fully respect the intrinsic characteristics of events streams. Given that event cameras only encode intensity changes and polarity rather than color intensities, estimating optical flow from events is arguably more difficult than from RGB information. We therefore propose to incorporate RGB information in an event-guided optical flow refinement strategy. Moreover, in light of the quasi-continuous nature of the time signals provided by event cameras, we propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages rather than in a single, long stage. Extensive experiments on both synthetic and real-world datasets show that these modifications lead to more reliable and realistic intermediate frame results than previous video frame interpolation methods. Our findings underline that a careful consideration of event characteristics such as high temporal density and elevated noise benefits interpolation accuracy.

Details

NeurIPS Conference 2022 Conference Paper

Teach Less, Learn More: On the Undistillable Classes in Knowledge Distillation

  • Yichen Zhu
  • Ning Liu
  • Zhiyuan Xu
  • Xin Liu
  • Weibin Meng
  • Louis Wang
  • Zhicai Ou
  • Jian Tang

Knowledge distillation (KD) can effectively compress neural networks by training a smaller network (student) to simulate the behavior of a larger one (teacher). A counter-intuitive observation is that a more expansive teacher does not make a better student, but the reasons for this phenomenon remain unclear. In this paper, we demonstrate that this is directly attributed to the presence of \textit{undistillable classes}: when trained with distillation, the teacher's knowledge of some classes is incomprehensible to the student model. We observe that while KD improves the overall accuracy, it is at the cost of the model becoming inaccurate in these undistillable classes. After establishing their widespread existence in state-of-the-art distillation methods, we illustrate their correlation with the capacity gap between teacher and student models. Finally, we present a simple Teach Less Learn More (TLLM) framework to identify and discard the undistillable classes during training. We validate the effectiveness of our approach on multiple datasets with varying network architectures. In all settings, our proposed method is able to exceed the performance of competitive state-of-the-art techniques.

PDF Details

IJCAI Conference 2019 Conference Paper

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

  • Weibin Meng
  • Ying Liu
  • Yichen Zhu
  • Shenglin Zhang
  • Dan Pei
  • Yuqing Liu
  • Yihao Chen
  • Ruizhi Zhang

Recording runtime status via logs is common for almost every computer system, and detecting anomalies in logs is crucial for timely identifying malfunctions of systems. However, manually detecting anomalies for logs is time-consuming, error-prone, and infeasible. Existing automatic log anomaly detection approaches, using indexes rather than semantics of log templates, tend to cause false alarms. In this work, we propose LogAnomaly, a framework to model unstructured a log stream as a natural language sequence. Empowered by template2vec, a novel, simple yet effective method to extract the semantic information hidden in log templates, LogAnomaly can detect both sequential and quantitive log anomalies simultaneously, which were not done by any previous work. Moreover, LogAnomaly can avoid the false alarms caused by the newly appearing log templates between periodic model retrainings. Our evaluation on two public production log datasets show that LogAnomaly outperforms existing log-based anomaly detection methods.

PDF Details

ICML Conference 2018 Conference Paper

Nonoverlap-Promoting Variable Selection

  • Pengtao Xie
  • Hongbao Zhang
  • Yichen Zhu
  • Eric P. Xing

Variable selection is a classic problem in machine learning (ML), widely used to find important explanatory factors, and improve generalization performance and interpretability of ML models. In this paper, we consider variable selection for models where multiple responses are to be predicted based on the same set of covariates. Since each response is relevant to a unique subset of covariates, we desire the selected variables for different responses have small overlap. We propose a regularizer that simultaneously encourage orthogonality and sparsity, which jointly brings in an effect of reducing overlap. We apply this regularizer to four model instances and develop efficient algorithms to solve the regularized problems. We provide a formal analysis on why the proposed regularizer can reduce generalization error. Experiments on both simulation studies and real-world datasets demonstrate the effectiveness of the proposed regularizer in selecting less-overlapped variables and improving generalization performance.

Details

ICML Conference 2018 Conference Paper

Orthogonality-Promoting Distance Metric Learning: Convex Relaxation and Theoretical Analysis

  • Pengtao Xie
  • Wei Wu 0023
  • Yichen Zhu
  • Eric P. Xing

Distance metric learning (DML), which learns a distance metric from labeled "similar" and "dissimilar" data pairs, is widely utilized. Recently, several works investigate orthogonality-promoting regularization (OPR), which encourages the projection vectors in DML to be close to being orthogonal, to achieve three effects: (1) high balancedness – achieving comparable performance on both frequent and infrequent classes; (2) high compactness – using a small number of projection vectors to achieve a "good" metric; (3) good generalizability – alleviating overfitting to training data. While showing promising results, these approaches suffer three problems. First, they involve solving non-convex optimization problems where achieving the global optimal is NP-hard. Second, it lacks a theoretical understanding why OPR can lead to balancedness. Third, the current generalization error analysis of OPR is not directly on the regularizer. In this paper, we address these three issues by (1) seeking convex relaxations of the original nonconvex problems so that the global optimal is guaranteed to be achievable; (2) providing a formal analysis on OPR’s capability of promoting balancedness; (3) providing a theoretical analysis that directly reveals the relationship between OPR and generalization performance. Experiments on various datasets demonstrate that our convex methods are more effective in promoting balancedness, compactness, and generalization, and are computationally more efficient, compared with the nonconvex methods.

Details
v2026.09.13