Author name cluster

Changyu Chen

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

2 author rows

ICLR Conference 2025 Conference Paper

Bootstrapping Language Models with DPO Implicit Rewards

Changyu Chen
Zichen Liu
Chao Du
Tianyu Pang
Qian Liu 0033
Arunesh Sinha
Pradeep Varakantham
Min Lin

Human alignment in large language models (LLMs) is an active area of research. A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO, after training, provides an implicit reward model. In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM. Our approach is to use the rewards from a current LLM to construct a preference dataset, which is then used in subsequent DPO rounds. We incorporate two refinements to further improve our approach: 1) length-regularized reward shaping to make the preference dataset length-unbiased; 2) experience replay to enhance the quality of the preference dataset. Our approach, named self-alignment with DPO ImpliCit rEwards (DICE), shows great improvements in alignment. It achieves an increase of more than 8$\\%$ in lengthcontrolled win rate on AlpacaEval 2 for all the different base models that we tried, without relying on external feedback. Our code is available at https://github.com/sail-sg/dice.

Details

AAMAS Conference 2025 Conference Paper

On Learning Informative Trajectory Embeddings for Imitation, Classification and Regression

Zichang Ge
Changyu Chen
Arunesh Sinha
Pradeep Varakantham

In real-world sequential decision making tasks like autonomous driving, robotics, and healthcare, learning from observed stateaction trajectories is critical for tasks like imitation, classification, and clustering. For example, self-driving cars must replicate human driving behaviors, while robots and healthcare systems benefit from modeling decision sequences, whether or not they come from expert data. Existing trajectory encoding methods often focus on specific tasks or rely on reward signals, limiting their ability to generalize across domains and tasks. Inspired by the success of embedding models like CLIP and BERT in static domains, we propose a novel method for embedding state-action trajectories into a latent space that captures the skills and competencies in the dynamic underlying decision-making processes. This method operates without the need for reward labels, enabling better generalization across diverse domains and tasks. Our contributions are threefold: (1) We introduce a trajectory embedding approach that captures multiple abilities from state-action data. (2) The learned embeddings exhibit strong representational power across downstream tasks, including imitation, classification, clustering, and regression. (3) The embeddings demonstrate unique properties, such as controlling agent behaviors in IQ-Learn and an additive structure in the latent space. Experimental results confirm that our method outperforms traditional approaches, offering more flexible and powerful trajectory representations for various applications. Our code is available at https: //github. com/Erasmo1015/vte.

PDF

EAAI Journal 2025 Journal Article

Planning scheme of artificial assembly posture and arm movement path in narrow space

Yizhen Zheng
Yuefeng Li
Xudong Pan
Fanwei Meng
Changyu Chen

Manual assembly in a narrow space involves problems of low efficiency and difficult assembly. In view of the lack of assembly process planning and assisted manual assembly in this kind of scenario, a hybrid modeling simulation method of human posture was proposed. This method combined the characteristics of manual assembly in narrow space. The assembly planning process was divided into two parts: trunk and lower limb posture planning and human arm movement planning, to reduce the complexity of planning and the difficulty of manual assembly. In the posture planning part, this study solved for the human trunk and lower limbs by establishing a multi-objective optimization model and achieved automatic screening of assembly posture according to the weight of each target element. Arm movement planning involved a neural network of assembly spaces to guide the sampling process of the path planner combined with the inverse solution of arm kinematics for environmental collision detection to quickly obtain a feasible collision-free arm movement path from the initial position to the assembly target. Finally, the feasibility of the method in a narrow space was verified by building a scene and carrying out the corresponding manual assembly operation experiments.

Details DOI

NeurIPS Conference 2024 Conference Paper

Uncovering Safety Risks of Large Language Models through Concept Activation Vector

Zhihao Xu
Ruixuan Huang
Changyu Chen
Xiting Wang

Despite careful safety alignment, current large language models (LLMs) remain vulnerable to various attacks. To further unveil the safety risks of LLMs, we introduce a Safety Concept Activation Vector (SCAV) framework, which effectively guides the attacks by accurately interpreting LLMs' safety mechanisms. We then develop an SCAV-guided attack method that can generate both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. Both automatic and human evaluations demonstrate that our attack method significantly improves the attack success rate and response quality while requiring less training data. Additionally, we find that our generated attack prompts may be transferable to GPT-4, and the embedding-level attacks may also be transferred to other white-box LLMs whose parameters are known. Our experiments further uncover the safety risks present in current LLMs. For example, in our evaluation of seven open-source LLMs, we observe an average attack success rate of 99. 14%, based on the classic keyword-matching criterion. Finally, we provide insights into the safety mechanism of LLMs. The code is available at https: //github. com/SproutNan/AI-Safety_SCAV.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning

Changyu Chen
Ramesha Karunasena
Thanh Nguyen
Arunesh Sinha
Pradeep Varakantham

Many problems in Reinforcement Learning (RL) seek an optimal policy with large discrete multidimensional yet unordered action spaces; these include problems in randomized allocation of resources such as placements of multiple security resources and emergency response units, etc. A challenge in this setting is that the underlying action space is categorical (discrete and unordered) and large, for which existing RL methods do not perform well. Moreover, these problems require validity of the realized action (allocation); this validity constraint is often difficult to express compactly in a closed mathematical form. The allocation nature of the problem also prefers stochastic optimal policies, if one exists. In this work, we address these challenges by (1) applying a (state) conditional normalizing flow to compactly represent the stochastic policy — the compactness arises due to the network only producing one sampled action and the corresponding log probability of the action, which is then used by an actor-critic method; and (2) employing an invalid action rejection method (via a valid action oracle) to update the base policy. The action rejection is enabled by a modified policy gradient that we derive. Finally, we conduct extensive experiments to show the scalability of our approach compared to prior methods and the ability to enforce arbitrary state-conditional constraints on the support of the distribution of actions in any state.

PDF Details

ICML Conference 2023 Conference Paper

Semi-Offline Reinforcement Learning for Optimized Text Generation

Changyu Chen
Xiting Wang
Yiqiao Jin
Victor Ye Dong
Li Dong
Jie Cao
Yi Liu
Rui Yan 0001

Existing reinforcement learning (RL) mainly utilize online or offline settings. The online methods explore the environment with expensive time cost, and the offline methods efficiently obtain reward signals by sacrificing the exploration capability. We propose semi-offline RL, a novel paradigm that can smoothly transit from the offline setting to the online setting, balances the exploration capability and training cost, and provides a theoretical foundation for comparing different RL settings. Based on the semi-offline MDP formulation, we present the RL setting that is optimal in terms of optimization cost, asymptotic error, and overfitting error bound. Extensive experiments show that our semi-offline RL approach is effective in various text generation tasks and datasets, and yields comparable or usually better performance compared with the state-of-the-art methods.

Details

AAAI Conference 2022 Conference Paper

Multiscale Generative Models: Improving Performance of a Generative Model Using Feedback from Other Dependent Generative Models

Changyu Chen
Avinandan Bose
Shih-Fen Cheng
Arunesh Sinha

Realistic fine-grained multi-agent simulation of real-world complex systems is crucial for many downstream tasks such as reinforcement learning. Recent work has used generative models (GANs in particular) for providing high-fidelity simulation of real-world systems. However, such generative models are often monolithic and miss out on modeling the interaction in multi-agent systems. In this work, we take a first step towards building multiple interacting generative models (GANs) that reflects the interaction in real world. We build and analyze a hierarchical set-up where a higher-level GAN is conditioned on the output of multiple lower-level GANs. We present a technique of using feedback from the higherlevel GAN to improve performance of lower-level GANs. We mathematically characterize the conditions under which our technique is impactful, including understanding the transfer learning nature of our set-up. We present three distinct experiments on synthetic data, time series data, and image domain, revealing the wide applicability of our technique.

PDF Details