Author name cluster

Tongran Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

12 papers

2 author rows

AAAI Conference 2026 Conference Paper

Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

Chenglong Wang
Yifu Huo
Yang Gan
Yongyu Mu
Qiaozhi He
Murun Yang
Bei Li
Chunliang Zhang

Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with LLM alignment performance, supporting it as a reliable reference for developing advanced reward models. By analyzing the evaluation results on MRMBench, we reveal that reward models struggle to simultaneously capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Furthermore, our results demonstrate that the proposed inference-time probing method provides a reliable metric for assessing the confidence of reward predictions, leading to improved alignment of large language models.

PDF Details DOI

YNIMG Journal 2026 Journal Article

The effects of working memory and interference control on reinforcement learning: Evidence from computational modeling and structural MRI

Mengxin Wen
Chengyan Yang
Tongran Liu
Kristoffer H. Madsen
Xun Liu

Reinforcement learning (RL) is crucial for adaptive decision-making in dynamic environments. Although prior studies have investigated the impact of working memory (WM) on RL, the joint contributions of WM and interference control (IC) to RL, as well as their underlying brain morphology, remain unclear. Here, 169 healthy young adults completed a probabilistic RL task manipulating WM load (low vs. high) and interference conditions (no interference vs. with interference), yielding measures of learning accuracy and key RL model parameters, including positive learning rate, inverse temperature, and forgetting parameter. A subset of 144 participants additionally underwent structural MRI to assess gray matter volume (GMV), cortical thickness, and sulcal depth. Results revealed that increasing WM load impaired learning accuracy and was accompanied by reduced learning rates and slower forgetting, with these effects associated with GMV, cortical thickness, and sulcal depth in limbic and frontoparietal control networks. Interference exposure decreased learning rates and accelerated forgetting, with IC-related differences in learning accuracy linked to sulcal depth in the cingulate gyrus. Notably, WM and IC interacted during learning, such that interference exposure selectively impaired learning accuracy under high WM load. In contrast, test accuracy was independently modulated by WM, indicating a dissociation between learning- and test-phase effects. The learning-phase interaction was further associated with thalamic GMV and sulcal depth in the temporal pole. These findings demonstrate how WM and IC jointly shape learning and emphasize the contributions of cortical and subcortical structural features to individual differences in learning.

Details DOI

ICML Conference 2025 Conference Paper

GRAM: A Generative Foundation Reward Model for Reward Generalization

Chenglong Wang 0002
Yang Gan
Yifu Huo
Yongyu Mu
Qiaozhi He
Murun Yang
Bei Li
Tong Xiao 0001

In aligning large language models (LLMs), reward models have played an important role, but are standardly trained as discriminative models and rely only on labeled human preference data. In this paper, we explore methods that train reward models using both unlabeled and labeled data. Building on the generative models in LLMs, we develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning. We also show that by using label smoothing, we are in fact optimizing a regularized pairwise ranking loss. This result, in turn, provides a new view of training reward models, which links generative models and discriminative models under the same class of training objectives. The outcome of these techniques is a foundation reward model, which can be applied to a wide range of tasks with little or no further fine-tuning effort. Extensive experiments show that this model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning, achieving significant performance improvements over several strong baseline models.

Details

NeurIPS Conference 2025 Conference Paper

MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization

Chenglong Wang
Yang Gan
Hang Zhou
Chi Hu
Yongyu Mu
Kai Song
Murun Yang
Bei Li

Recent advances in diffusion language models (DLMs) have presented a promising alternative to traditional autoregressive large language models (LLMs). However, DLMs still lag behind LLMs in reasoning performance, especially as the number of denoising steps decreases. Our analysis reveals that this shortcoming arises primarily from the independent generation of masked tokens across denoising steps, which fails to capture the token correlation. In this paper, we define two types of token correlation: intra-sequence correlation and inter-sequence correlation, and demonstrate that enhancing these correlations improves reasoning performance. To this end, we propose a Multi-Reward Optimization (MRO) approach, which encourages DLMs to consider the token correlation during the denoising process. More specifically, our MRO approach leverages test-time scaling, reject sampling, and reinforcement learning to directly optimize the token correlation with multiple elaborate rewards. Additionally, we introduce group step and importance sampling strategies to mitigate reward variance and enhance sampling efficiency. Through extensive experiments, we demonstrate that MRO not only improves reasoning performance but also achieves significant sampling speedups while maintaining high performance on reasoning benchmarks.

PDF Details

AAAI Conference 2025 Conference Paper

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

Chenglong Wang
Yang Gan
Yifu Huo
Yongyu Mu
Murun Yang
Qiaozhi He
Tong Xiao
Chunliang Zhang

Large vision-language models (LVLMs) often fail to align with human preferences, leading to issues like generating misleading content without proper visual context (also known as hallucination). A promising solution to this problem is using human-preference alignment techniques, such as best-of-n sampling and reinforcement learning. However, these techniques face the difficulty arising from the scarcity of visual preference data, which is required to train a visual reward model (VRM). In this work, we continue the line of research. We present a Robust Visual Reward Model (RoVRM) which improves human-preference alignment for LVLMs. RoVRM leverages auxiliary textual preference data through a three-phase progressive training and optimal transport-based preference data selection to effectively mitigate the scarcity of visual preference data. We experiment with RoVRM on the commonly used vision-language tasks based on the LLaVA-1.5-7B and -13B models. Experimental results demonstrate that RoVRM consistently outperforms traditional VRMs. Furthermore, our three-phase progressive training and preference data selection approaches can yield consistent performance gains over ranking-based alignment techniques, such as direct preference optimization.

PDF Details DOI

AAAI Conference 2024 Conference Paper

ESRL: Efficient Sampling-Based Reinforcement Learning for Sequence Generation

Chenglong Wang
Hang Zhou
Yimin Hu
Yifu Huo
Bei Li
Tongran Liu
Tong Xiao
Jingbo Zhu

Applying Reinforcement Learning (RL) to sequence generation models enables the direct optimization of long-term rewards (e.g., BLEU and human feedback), but typically requires large-scale sampling over a space of action sequences. This is a computational challenge as presented by the practice of sequence generation problems, such as machine translation, where we often deal with a large action space (e.g., a vocabulary) and a long action sequence (e.g., a translation). In this work, we introduce two-stage sampling and dynamic sampling approaches to improve the sampling efficiency during training sequence generation models via RL. We experiment with our approaches on the traditional sequence generation tasks, including machine translation and abstractive summarization. Furthermore, we evaluate our approaches in RL from human feedback (RLHF) through training a large language model using the reward model. Experimental results show that the efficient sampling-based RL, referred to as ESRL, can outperform all baselines in terms of both training efficiency and memory consumption. Notably, ESRL yields consistent performance gains over the strong REINFORCE, minimum risk training, and proximal policy optimization methods. The code is available at https://github.com/wangclnlp/DeepSpeed-Chat-Extension/examples/esrl.

PDF Details DOI

AAAI Conference 2020 Conference Paper

Neural Machine Translation with Joint Representation

Yanyang Li
Qiang Wang
Tong Xiao
Tongran Liu
Jingbo Zhu

Though early successes of Statistical Machine Translation (SMT) systems are attributed in part to the explicit modelling of the interaction between any two source and target units, e. g. , alignment, the recent Neural Machine Translation (NMT) systems resort to the attention which partially encodes the interaction for efﬁciency. In this paper, we employ Joint Representation that fully accounts for each possible interaction. We sidestep the inefﬁciency issue by reﬁning representations with the proposed efﬁcient attention operation. The resulting Reformer models offer a new Sequence-to- Sequence modelling paradigm besides the Encoder-Decoder framework and outperform the Transformer baseline in either the small scale IWSLT14 German-English, English-German and IWSLT15 Vietnamese-English or the large scale NIST12 Chinese-English translation tasks by about 1 BLEU point. We also propose a systematic model scaling approach, allowing the Reformer model to beat the state-of-the-art Transformer in IWSLT14 German-English and NIST12 Chinese-English with about 50% fewer parameters. The code is publicly available at https: //github. com/lyy1994/reformer.

PDF Details

IJCAI Conference 2020 Conference Paper

Towards Fully 8-bit Integer Inference for the Transformer Model

Ye Lin
Yanyang Li
Tengbo Liu
Tong Xiao
Tongran Liu
Jingbo Zhu

8-bit integer inference, as a promising direction in reducing both the latency and storage of deep neural networks, has made great progress recently. On the other hand, previous systems still rely on 32-bit floating point for certain functions in complex models (e. g. , Softmax in Transformer), and make heavy use of quantization and de-quantization. In this work, we show that after a principled modification on the Transformer architecture, dubbed Integer Transformer, an (almost) fully 8-bit integer inference algorithm Scale Propagation could be derived. De-quantization is adopted when necessary, which makes the network more efficient. Our experiments on WMT16 En Ro, WMT14 En De and En->Fr translation tasks as well as the WikiText-103 language modelling task show that the fully 8-bit Transformer system achieves comparable performance with the floating point baseline but requires nearly 4x less memory footprint.

PDF Details DOI

IJCAI Conference 2019 Conference Paper

Sharing Attention Weights for Fast Transformer

Tong Xiao
Yinqiao Li
Jingbo Zhu
Zhengtao Yu
Tongran Liu

Recently, the Transformer machine translation system has shown strong results by stacking attention layers on both the source and target-language sides. But the inference of this model is slow due to the heavy use of dot-product attention in auto-regressive decoding. In this paper we speed up Transformer via a fast and lightweight attention model. More specifically, we share attention weights in adjacent layers and enable the efficient re-use of hidden states in a vertical manner. Moreover, the sharing policy can be jointly learned with the MT model. We test our approach on ten WMT and NIST OpenMT tasks. Experimental results show that it yields an average of 1. 3X speed-up (with almost no decrease in BLEU) on top of a state-of-the-art implementation that has already adopted a cache for fast inference. Also, our approach obtains a 1. 8X speed-up when it works with the AAN model. This is even 16 times faster than the baseline with no use of the attention cache.

PDF Details

IJCAI Conference 2017 Conference Paper

Fast Parallel Training of Neural Language Models

Tong Xiao
Jingbo Zhu
Tongran Liu
Chunliang Zhang

Training neural language models (NLMs) is very time consuming and we need parallelization for system speedup. However, standard training methods have poor scalability across multiple devices (e. g. , GPUs) due to the huge time cost required to transmit data for gradient sharing in the back-propagation process. In this paper we present a sampling-based approach to reducing data transmission for better scaling of NLMs. As a ''bonus'', the resulting model also improves the training speed on a single device. Our approach yields significant speed improvements on a recurrent neural network-based language model. On four NVIDIA GTX1080 GPUs, it achieves a speedup of 2. 1+ times over the standard asynchronous stochastic gradient descent baseline, yet with no increase in perplexity. This is even 4. 2 times faster than the naive single GPU counterpart.

PDF Details

AAAI Conference 2016 Conference Paper

Syntactic Skeleton-Based Translation

Tong Xiao
Jingbo Zhu
Chunliang Zhang
Tongran Liu

In this paper we propose an approach to modeling syntactically-motivated skeletal structure of source sentence for machine translation. This model allows for application of high-level syntactic transfer rules and low-level non-syntactic rules. It thus involves fully syntactic, non-syntactic, and partially syntactic derivations via a single grammar and decoding paradigm. On large-scale Chinese-English and English- Chinese translation tasks, we obtain an average improvement of +0. 9 BLEU across the newswire and web genres.

PDF Details

AIJ Journal 2013 Journal Article

Bagging and Boosting statistical machine translation systems

Tong Xiao
Jingbo Zhu
Tongran Liu

In this article we address the issue of generating diversified translation systems from a single Statistical Machine Translation (SMT) engine for system combination. Unlike traditional approaches, we do not resort to multiple structurally different SMT systems, but instead directly learn a strong SMT system from a single translation engine in a principled way. Our approach is based on Bagging and Boosting which are two instances of the general framework of ensemble learning. The basic idea is that we first generate an ensemble of weak translation systems using a base learning algorithm, and then learn a strong translation system from the ensemble. One of the advantages of our approach is that it can work with any of current SMT systems and make them stronger almost “for free”. Beyond this, most system combination methods are directly applicable to the proposed framework for generating the final translation system from the ensemble of weak systems. We evaluate our approach on Chinese–English translation in three state-of-the-art SMT systems, including a phrase-based system, a hierarchical phrase-based system and a syntax-based system. Experimental results on the NIST MT evaluation corpora show that our approach leads to significant improvements in translation accuracy over the baselines. More interestingly, it is observed that our approach is able to improve the existing system combination systems. The biggest improvements are obtained by generating weak systems using Bagging/Boosting, and learning the strong system using a state-of-the-art system combination method.

Details DOI