Author name cluster

Huayu Chen

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

14 papers

2 author rows

ICML Conference 2025 Conference Paper

Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator

Kaiwen Zheng
Yongxin Chen
Huayu Chen
Guande He
Ming-Yu Liu 0001
Jun Zhu 0001
Qinsheng Zhang

While likelihood-based generative models, particularly diffusion and autoregressive models, have achieved remarkable fidelity in visual generation, the maximum likelihood estimation (MLE) objective, which minimizes the forward KL divergence, inherently suffers from a mode-covering tendency that limits the generation quality under limited model capacity. In this work, we propose Direct Discriminative Optimization (DDO) as a unified framework that integrates likelihood-based generative training and GAN-type discrimination to bypass this fundamental constraint by exploiting reverse KL and self-generated negative signals. Our key insight is to parameterize a discriminator implicitly using the likelihood ratio between a learnable target model and a fixed reference model, drawing parallels with the philosophy of Direct Preference Optimization (DPO). Unlike GANs, this parameterization eliminates the need for joint training of generator and discriminator networks, allowing for direct, efficient, and effective finetuning of a well-trained model to its full potential beyond the limits of MLE. DDO can be performed iteratively in a self-play manner for progressive model refinement, with each round requiring less than 1% of pretraining epochs. Our experiments demonstrate the effectiveness of DDO by significantly advancing the previous SOTA diffusion model EDM, reducing FID scores from 1. 79/1. 58/1. 96 to new records of 1. 30/0. 97/1. 26 on CIFAR-10/ImageNet-64/ImageNet 512$\times$512 datasets without any guidance mechanisms, and by consistently improving both guidance-free and CFG-enhanced FIDs of visual autoregressive models on ImageNet 256$\times$256.

Details

ICML Conference 2025 Conference Paper

Free Process Rewards without Process Labels

Lifan Yuan
Wendi Li
Huayu Chen
Ganqu Cui
Ning Ding 0002
Kaiyan Zhang
Bowen Zhou 0002
Zhiyuan Liu 0001

Different from its counterpart outcome reward models (ORMs), which evaluate the entire responses, a process reward model (PRM) scores a reasoning trajectory step by step, providing denser and more fine-grained rewards. However, training a PRM requires labels annotated at every intermediate step, presenting significant challenges for both manual and automatic data collection. This paper aims to address this challenge. Both theoretically and empirically, we show that an implicit PRM can be obtained at no additional cost, by simply training an ORM on the cheaper response-level labels. The only assumption is to parameterize the outcome reward as the log-likelihood ratios of the policy and reference models r$\phi$(y) = $\beta$ log $\pi$$\phi$(y) $\pi$ref(y), which can be optimized regardless of the specific choice of loss objectives. In experiments, we instantiate our implicit PRMs with various objectives and evaluate their performance on MATH. We show that our implicit PRM outperforms a strong MCTS-based baseline á la Math-Shepherd (Wang et al. , 2023) using less than 1/38 of the training data. Its performance can be further improved with majority voting. We further find that scaling up instructions and responses benefits our implicit PRM, and the latter brings a larger gain. Particularly, we find that our implicit PRM, when instantiated with the cross-entropy (CE) loss, is more data-efficient and can keep improving generation models even when trained with only one response per instruction, the setup that suffers from extreme data scarcity and imbalance. Further, instructions should be relevant to downstream tasks while the diversity of responses does not bring gains. Surprisingly, training on extra Math-Shepherd step labels brings no further improvements to our implicit PRM trained on only outcome data. We hope that our work will encourage a rethinking of PRM training approaches and contribute to making training PRMs more accessible.

Details

ICLR Conference 2025 Conference Paper

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

Songming Liu
Lingxuan Wu
Bangguo Li
Hengkai Tan
Huayu Chen
Zhengyi Wang
Ke Xu
Hang Su 0006

Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds on diffusion models to effectively represent multi-modality, with innovative designs of a scalable Transformer to deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, we managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to $1.2$B parameters, which is the largest diffusion-based foundation model for robotic manipulation. We finally fine-tuned RDT on a self-created multi-task bimanual dataset with over $6$K+ episodes to refine its manipulation capabilities. Experiments on real robots demonstrate that RDT significantly outperforms existing methods. It exhibits zero-shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just 1$\sim$5 demonstrations, and effectively handles complex, dexterous tasks. We refer to https://rdt-robotics.github.io/rdt-robotics/ for the code and videos.

Details

ICLR Conference 2025 Conference Paper

Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignment

Huayu Chen
Hang Su 0006
Peize Sun
Jun Zhu 0001

Classifier-Free Guidance (CFG) is a critical technique for enhancing the sample quality of visual generative models. However, in autoregressive (AR) multi-modal generation, CFG introduces design inconsistencies between language and visual content, contradicting the design philosophy of unifying different modalities for visual AR. Motivated by language model alignment methods, we propose Condition Contrastive Alignment (CCA) to facilitate guidance-free AR visual generation. Unlike guidance methods that alter the sampling process to achieve the ideal sampling distribution, CCA directly fine-tunes pretrained models to fit the same distribution target. Experimental results show that CCA can significantly enhance the guidance-free performance of all tested models with just one epoch of fine-tuning (1% of pretraining epochs) on the pretraining dataset. This largely removes the need for guided sampling in AR visual generation and cuts the sampling cost by half. Moreover, by adjusting training parameters, CCA can achieve trade-offs between sample diversity and fidelity similar to CFG. This experimentally confirms the strong theoretical connection between language-targeted alignment and visual-targeted guidance methods, unifying two previously independent research fields.

Details

ICML Conference 2025 Conference Paper

Visual Generation Without Guidance

Huayu Chen
Kai Jiang
Kaiwen Zheng
Jianfei Chen 0001
Hang Su 0006
Jun Zhu 0001

Classifier-Free Guidance (CFG) has been a default technique in various visual generative models, yet it requires inference from both conditional and unconditional models during sampling. We propose to build visual models that are free from guided sampling. The resulting algorithm, Guidance-Free Training (GFT), matches the performance of CFG while reducing sampling to a single model, halving the computational cost. Unlike previous distillation-based approaches that rely on pretrained CFG networks, GFT enables training directly from scratch. GFT is simple to implement. It retains the same maximum likelihood objective as CFG and differs mainly in the parameterization of conditional models. Implementing GFT requires only minimal modifications to existing codebases, as most design choices and hyperparameters are directly inherited from CFG. Our extensive experiments across five distinct visual models demonstrate the effectiveness and versatility of GFT. Across domains of diffusion, autoregressive, and masked-prediction modeling, GFT consistently achieves comparable or even lower FID scores, with similar diversity-fidelity trade-offs compared with CFG baselines, all while being guidance-free.

Details

NeurIPS Conference 2024 Conference Paper

Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control

Huayu Chen
Kaiwen Zheng
Hang Su
Jun Zhu

Drawing upon recent advances in language model alignment, we formulate offline Reinforcement Learning as a two-stage optimization problem: First pretraining expressive generative policies on reward-free behavior datasets, then finetuning these policies to align with task-specific annotations like Q-values. This strategy allows us to leverage abundant and diverse behavior data to enhance generalization and enable rapid adaptation to downstream tasks using minimal annotations. In particular, we introduce Efficient Diffusion Alignment (EDA) for solving continuous control problems. EDA utilizes diffusion models for behavior modeling. However, unlike previous approaches, we represent diffusion policies as the derivative of a scalar neural network with respect to action inputs. This representation is critical because it enables direct density calculation for diffusion models, making them compatible with existing LLM alignment theories. During policy fine-tuning, we extend preference-based alignment methods like Direct Preference Optimization (DPO) to align diffusion behaviors with continuous Q-functions. Our evaluation on the D4RL benchmark shows that EDA exceeds all baseline methods in overall performance. Notably, EDA maintains about 95\% of performance and still outperforms several baselines given only 1\% of Q-labelled data during fine-tuning.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

Tianjiao Luo
Tim Pearce
Huayu Chen
Jianfei Chen
Jun Zhu

Generative Adversarial Imitation Learning (GAIL) provides a promising approach to training a generative policy to imitate a demonstrator. It uses on-policy Reinforcement Learning (RL) to optimize a reward signal derived from an adversarial discriminator. However, optimizing GAIL is difficult in practise, with the training loss oscillating during training, slowing convergence. This optimization instability can prevent GAIL from finding a good policy, harming its final performance. In this paper, we study GAIL’s optimization from a control-theoretic perspective. We show that GAIL cannot converge to the desired equilibrium. In response, we analyze the training dynamics of GAIL in function space and design a novel controller that not only pushes GAIL to the desired equilibrium but also achieves asymptotic stability in a simplified “one-step” setting. Going from theory to practice, we propose Controlled-GAIL (C-GAIL), which adds a differentiable regularization term on the GAIL objective to stabilize training. Empirically, the C-GAIL regularizer improves the training of various existing GAIL methods, including the popular GAIL-DAC, by speeding up the convergence, reducing the range of oscillation, and matching the expert distribution more closely.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Noise Contrastive Alignment of Language Models with Explicit Rewards

Huayu Chen
Guande He
Lifan Yuan
Ganqu Cui
Hang Su
Jun Zhu

User intentions are typically formalized as evaluation rewards to be maximized when fine-tuning language models (LMs). Existing alignment methods, such as Direct Preference Optimization (DPO), are mainly tailored for pairwise preference data where rewards are implicitly defined rather than explicitly given. In this paper, we introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. Our framework comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data. Notably, we show that the DPO loss is a special case of our proposed InfoNCA objective under pairwise preference settings, thereby integrating and extending current alignment theories. By comparing NCA and InfoNCA, we demonstrate that the well-observed decreasing-likelihood trend of DPO/InfoNCA is caused by their focus on adjusting relative likelihood across different responses. In contrast, NCA optimizes the absolute likelihood for each response, thereby effectively preventing the chosen likelihood from decreasing. We evaluate our methods in both reward and preference settings with Mistral-8$\times$7B and 7B models. Experiments suggest that InfoNCA/NCA surpasses various preference baselines when reward datasets are available. We also find NCA significantly outperforms DPO in complex reasoning tasks like math and coding.

PDF Details DOI

ICLR Conference 2024 Conference Paper

Score Regularized Policy Optimization through Diffusion Behavior

Huayu Chen
Cheng Lu 0011
Zhengyi Wang
Hang Su 0006
Jun Zhu 0001

Recent developments in offline reinforcement learning have uncovered the immense potential of diffusion modeling, which excels at representing heterogeneous behavior policies. However, sampling from diffusion policies is considerably slow because it necessitates tens to hundreds of iterative inference steps for one action. To address this issue, we propose to extract an efficient deterministic inference policy from critic models and pretrained diffusion behavior models, leveraging the latter to directly regularize the policy gradient with the behavior distribution’s score function during optimization. Our method enjoys powerful generative capabilities of diffusion modeling while completely circumventing the computationally intensive and time-consuming diffusion sampling scheme, both during training and evaluation. Extensive results on D4RL tasks show that our method boosts action sampling speed by more than 25 times compared with various leading diffusion-based methods in locomotion tasks, while still maintaining state-of-the-art performance.

Details

JBHI Journal 2023 Journal Article

Clustering-Fusion Feature Selection Method in Identifying Major Depressive Disorder Based on Resting State EEG Signals

Shuting Sun
Huayu Chen
Gang Luo
Chang Yan
Qunxi Dong
Xuexiao Shao
Xiaowei Li
Bin Hu

Depression is a heterogeneous syndrome with certain individual differences among subjects. Exploring a feature selection method that can effectively mine the commonness intra-groups and the differences inter-groups in depression recognition is therefore of great significance. This study proposed a new clustering-fusion feature selection method. Hierarchical clustering (HC) algorithm was used to capture the heterogeneity distribution of subjects. Average and similarity network fusion (SNF) algorithms were adopted to characterize the brain network atlas of different populations. Differences analysis was also utilized to obtain the features with discriminant performance. Experiments showed that compared with traditional feature selection methods, HCSNF method yielded the optimal classification results of depression recognition in both sensor and source layers of electroencephalography (EEG) data. Especially in the beta band of EEG data at sensor layer, the classification performance was improved by more than 6%. Moreover, the long-distance connections between parietal-occipital lobe and other brain regions not only have high discriminative power, but also significantly correlate with depressive symptoms, indicating the important role of these features in depression recognition. Therefore, this study may provide methodological guidance for the discovery of reproducible electrophysiological biomarkers and new insights into common neuropathological mechanisms of heterogeneous depression diseases.

Details DOI

ICML Conference 2023 Conference Paper

Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning

Cheng Lu 0011
Huayu Chen
Jianfei Chen 0001
Hang Su 0006
Chongxuan Li
Jun Zhu 0001

Guided sampling is a vital approach for applying diffusion models in real-world tasks that embeds human-defined guidance during the sampling procedure. This paper considers a general setting where the guidance is defined by an (unnormalized) energy function. The main challenge for this setting is that the intermediate guidance during the diffusion sampling procedure, which is jointly defined by the sampling distribution and the energy function, is unknown and is hard to estimate. To address this challenge, we propose an exact formulation of the intermediate guidance as well as a novel training objective named contrastive energy prediction (CEP) to learn the exact guidance. Our method is guaranteed to converge to the exact guidance under unlimited model capacity and data samples, while previous methods can not. We demonstrate the effectiveness of our method by applying it to offline reinforcement learning (RL). Extensive experiments on D4RL benchmarks demonstrate that our method outperforms existing state-of-the-art algorithms. We also provide some examples of applying CEP for image synthesis to demonstrate the scalability of CEP on high-dimensional data.

Details

ICLR Conference 2023 Conference Paper

Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

Huayu Chen
Cheng Lu 0011
Chengyang Ying
Hang Su 0006
Jun Zhu 0001

In offline reinforcement learning, weighted regression is a common method to ensure the learned policy stays close to the behavior policy and to prevent selecting out-of-sample actions. In this work, we show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training, which deviates from their initial motivation. To address this problem, we adopt a generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. The key insight is that such decoupling avoids learning an explicitly parameterized policy model with a closed-form expression. Directly learning the behavior policy allows us to leverage existing advances in generative modeling, such as diffusion-based methods, to model diverse behaviors. As for action evaluation, we combine our method with an in-sample planning technique to further avoid selecting out-of-sample actions and increase computational efficiency. Experimental results on D4RL datasets show that our proposed method achieves competitive or superior performance compared with state-of-the-art offline RL methods, especially in complex tasks such as AntMaze. We also empirically demonstrate that our method can successfully learn from a heterogeneous dataset containing multiple distinctive but similarly successful strategies, whereas previous unimodal policies fail.

Details

YNICL Journal 2022 Journal Article

Increased cortical-medulla functional connectivity is correlated with swallowing in dysphagia patients with subacute infratentorial stroke

Meng Dai
Jia Qiao
Xiaomei Wei
Huayu Chen
Zhonghui Shi
Zulin Dou

Patients with infratentorial stroke (IS) exhibit more severe dysphagia and a higher risk of aspiration than patients with supratentorial stroke. Nevertheless, a large proportion of patients with IS regain swallowing function within 6 months; however, the neural mechanism for this recovery remains unclear. We aimed to investigate possible neuroplastic changes involved using functional magnetic resonance imaging (fMRI) and their relation to swallowing function. We assessed 21 patients with IS (mean age: 59.9 ± 11.1 years) exhibiting dysphagia in the subacute phase and 21 healthy controls (mean age: 57.1 ± 7.8 years). Patient evaluations were based on the functional oral intake scale (FOIS), videofluoroscopic swallow study (VFSS), and fMRI. Temporal swallowing measures and the penetration-aspiration scale (PAS) were obtained using VFSS. Whole-brain-medulla resting-state functional connectivity (rsFC) was calculated and compared between patients and healthy controls. The rsFCs were also correlated with functional measures within the patient group. In patients with IS, whole-brain-medulla rsFCs were significantly higher in the precuneus, the left and right precentral gyrus, and the right supplementary motor area compared to those in healthy controls (P < 0.001, family-wise error-corrected cluster-level P < 0.05). The rsFCs to the medulla for the left (r = -0.507, P = 0.027) and right side (r = -0.503, P = 0.028) precentral gyrus were negatively correlated with the PAS. The rsFC between the left (r = 0.470, P = 0.042) and right (r = 0.459, P = 0.048) precentral gyrus to the medulla was positively correlated with upper esophageal sphincter opening durations (UOD). In addition, PAS was also correlated with UOD (r = -0.638, P = 0.003) whereas the laryngeal closure duration was correlated with the hyoid bone movement duration (r = 0.550, P = 0.015). Patients with IS exhibited overall modulation of cortical-medulla connectivity during the subacute phase. Patients with higher connectivities showed better swallowing performance. These findings support that there is cortical involvement in swallowing regulation after IS and can aid in determining potential treatment targets for dysphagia.

Details DOI

JMLR Journal 2022 Journal Article

Tianshou: A Highly Modularized Deep Reinforcement Learning Library

Jiayi Weng
Huayu Chen
Dong Yan
Kaichao You
Alexis Duburcq
Minghao Zhang
Yi Su
Hang Su

In this paper, we present Tianshou, a highly modularized Python library for deep reinforcement learning (DRL) that uses PyTorch as its backend. Tianshou intends to be research-friendly by providing a flexible and reliable infrastructure of DRL algorithms. It supports online and offline training with more than 20 classic algorithms through a unified interface. To facilitate related research and prove Tianshou's reliability, we have released Tianshou's benchmark of MuJoCo environments, covering eight classic algorithms with state-of-the-art performance. We open-sourced Tianshou at https://github.com/thu-ml/tianshou/. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2022. ( edit, beta )

PDF Details