Author name cluster

Xingchun Diao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

2 papers

2 author rows

AAAI Conference 2026 Conference Paper

AIM: Manifold-based Data Filtering for Representation Finetuning

Qing Li
Qibin Zheng
Yi Liu
Xingchun Diao

Representation Finetuning (ReFT) has recently emerged as an efficient paradigm for adapting pretrained language models by editing hidden representations rather than model weights. However, our preliminary experiments reveal that ReFT is notably more sensitive to training data quality compared to traditional parameter-efficient finetuning methods, particularly to samples with incorrect labels, which can severely degrade performance. Inspired by prior work demonstrating that the hidden representations of generalizable neural networks exhibit low-dimensional manifold structures, we hypothesize that effective generalization in ReFT requires geometrically structured transformations between pre- and post-intervention representations. This implies that the intervention vectors representing these transformations should form a low-dimensional manifold, rendering the inconsistent transformations induced by label noise as detectable geometric outliers. To leverage this insight, we introduce Aligning Interventions on a learned Manifold (AIM), a representation-based data filtering method for ReFT, which identifies high-quality training samples by measuring the geometric consistency of their intervention vectors with respect to a robust reference manifold derived via principal component analysis on trusted data. Extensive experiments on both commonsense and arithmetic reasoning tasks confirm the effectiveness of AIM, showing consistent improvements over strong data selection baselines across multiple model scales.

PDF Details DOI

ECAI Conference 2025 Conference Paper

Don't Stop Pre-Training Small Language Models for Continual Enhancement of Reasoning

Qing Li
Qibin Zheng
Yi Liu 0043
Xingchun Diao

We investigate the continual enhancement of mathematical reasoning abilities in small language models (SLMs). While large language models (LLMs) demonstrate impressive reasoning performance, their deployment is often constrained by substantial computational costs. Existing approaches to improving SLMs mainly rely on knowledge distillation from costly teacher LLMs, which typically improves mathematical reasoning at the expense of general capabilities. In this work, we show that continual pre-training (CPT) has strong potential to enhance the mathematical reasoning ability of SLMs without relying on large teacher models. We also find that its effectiveness critically depends on the quality of the training data. To maximize efficiency and performance, we propose Dual-Metric Selection for Continual Pre-training (DRIFT), a novel data selection strategy that identifies optimal training data through task-aligned loss differences and distributional regularization. To further enhance task-specific reasoning while preserving general capabilities, we introduce a metadata-aware data mixture that integrates diverse sources during CPT. Extensive experiments on multiple arithmetic reasoning benchmarks demonstrate the effectiveness of DRIFT: SLMs trained with DRIFT achieve substantial gains in reasoning performance, surpassing larger models on specific tasks, while largely preserving general capabilities.

Details