Author name cluster

Jun Xie

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

14 papers

2 author rows

AAAI Conference 2026 Conference Paper

Dynamic Deep Graph Learning for Incomplete Multi-View Clustering with Masked Graph Reconstruction Loss

Zhenghao Zhang
Jun Xie
Xingchen Chen
Tao Yu
Hongzhu Yi
Kaixin Xu
Yuanxiang Wang
Tianyu Zong

The prevalence of real-world multi-view data makes incomplete multi-view clustering (IMVC) a crucial research. The rapid development of Graph Neural Networks (GNNs) has established them as one of the mainstream approaches for multi-view clustering. Despite significant progress in GNNs-based IMVC, some challenges remain: (1) Most methods rely on the K-Nearest Neighbors (KNN) algorithm to construct static graphs from raw data, which introduces noise and diminishes the robustness of the graph topology. (2) Existing methods typically utilize the Mean Squared Error (MSE) loss between the reconstructed graph and the sparse adjacency graph directly as the graph reconstruction loss, leading to substantial gradient noise during optimization. To address these issues, we propose a novel Dynamic Deep Graph Learning for Incomplete Multi-View Clustering with Masked Graph Reconstruction Loss (DGIMVCM). Firstly, we construct a missing-robust global graph from the raw data. A graph convolutional embedding layer is then designed to extract primary features and refined dynamic view-specific graph structures, leveraging the global graph for imputation of missing views. This process is complemented by graph structure contrastive learning, which identifies consistency among view-specific graph structures. Secondly, a graph self-attention encoder is introduced to extract high-level representations based on the imputed primary features and view-specific graphs, and is optimized with a masked graph reconstruction loss to mitigate gradient noise during optimization. Finally, a clustering module is constructed and optimized through a pseudo-label self-supervised training mechanism. Extensive experiments on multiple datasets validate the effectiveness and superiority of DGIMVCM.

PDF Details DOI

IROS Conference 2025 Conference Paper

Constrained Behavior Cloning for Robotic Learning

Jun Xie
Jianwei Tan
Wensheng Liang
Zhicheng Wang
Xiaoguang Ma

Behavior cloning (BC) is a widely used method for learning from expert demonstrations due to its simplicity and efficiency. However, the reliability and stability of BC decline when facing data distribution shifts, especially in single-arm robots with limited fields of view. This study introduces a Geometrically and Historically Constrained Behavior Cloning (GHCBC) method, where an HCBC module utilizes visual and action histories to capture temporal dependencies, maximizing the use of available information, and a GCBC module incorporates high-level perceptual data, such as the relative poses of joints and end-effectors, to enhance BC performance. Experiments demonstrate that the GHCBC outperforms current SOTA BC methods, achieving a 31. 5% improvement in simulation success rates and 48. 4% in real-robot scenarios respectively. To the best of our knowledge, this is the first time that the GHCBC has been introduced in robotic BC where great potential is demonstrated for long-term tasks in real world environments.

Details

ICML Conference 2025 Conference Paper

Deliberation in Latent Space via Differentiable Cache Augmentation

Luyang Liu
Jonas Pfeiffer
Jiaxing Wu
Jun Xie
Arthur Szlam

Techniques enabling large language models (LLMs) to "think more" by generating and attending to intermediate reasoning steps have shown promise in solving complex problems. However, the standard approaches generate sequences of discrete tokens immediately before responding, and so they can incur significant latency costs and be challenging to optimize. In this work, we demonstrate that a frozen LLM can be augmented with an offline coprocessor that operates on the model’s key-value (kv) cache. This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding. We train this coprocessor using the language modeling loss from the decoder on standard pretraining data, while keeping the decoder itself frozen. This approach enables the model to learn, in an end-to-end differentiable fashion, how to distill additional computation into its kv-cache. Because the decoder remains unchanged, the coprocessor can operate offline and asynchronously, and the language model can function normally if the coprocessor is unavailable or if a given cache is deemed not to require extra computation. We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens. Furthermore, even without any task-specific training, our experiments demonstrate that cache augmentation consistently improves performance across a range of reasoning-intensive tasks.

Details

AAAI Conference 2025 Conference Paper

RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs

Jiaxing Wu
Lin Ning
Luyang Liu
Harrison Lee
Neo Wu
Chao Wang
Sushant Prakash
Shawn O'Banion

LLM-powered personalization agent systems employ Large Language Models (LLMs) to predict users’ behavior from their past activities. However, their effectiveness often hinges on the ability to effectively leverage extensive, long user historical data due to its inherent noise and length of such data. Existing pre-trained LLMs may generate summaries that are concise but lack the necessary context for downstream tasks, hindering their utility in personalization systems. To address these challenges, we introduce Reinforcement Learning from Prediction Feedback (RLPF). RLPF fine-tunes LLMs to generate concise, human-readable user summaries that are optimized for downstream task performance. By maximizing the usefulness of the generated summaries, RLPF effectively distills extensive user history data while preserving essential information for downstream tasks. Our empirical evaluation demonstrates significant improvements in both extrinsic downstream task utility and intrinsic summary quality, surpassing baseline methods by up to 22% and achieving an up to 84.59% win rate on Factuality, Abstractiveness, and Readability. RLPF also achieves a remarkable 74% reduction while improving performance on 16 out of 19 unseen tasks and/or datasets, showcasing its generalizability. This approach offers a promising solution for enhancing LLM personalization by effectively transforming long, noisy user histories into informative and human-readable representations.

PDF Details DOI

ECAI Conference 2025 Conference Paper

Subconscious Robotic Imitation Learning

Jun Xie
Zhicheng Wang
Jianwei Tan
Huanxu Lin
Yang Jiang
Xiaoguang Ma

While imitation learning (IL) emerges as a promising paradigm for embodied intelligent robots, its practical application is constrained by slow execution speeds, caused by the computational intensity of precise multi-model trajectory prediction, especially in complex dynamic environments. In contrast, humans can efficiently perform long-duration tasks through subconscious-driven habitual actions, such as riding bikes, without focusing on execution details. Motivated by this insight, we proposed Subconscious Robotic Imitation Learning (SRIL) framework, which mimicked the subconscious information extraction and decision-making abilities through intent-aware sampling and cognitive hierarchical reasoning, thereby significantly improving IL task execution efficiency. Experimental results demonstrated that execution speeds of the SRIL were 100% to 200% faster over SOTA policies for comprehensive bimanual tasks, with consistently higher success rates.

Details

ECAI Conference 2024 Conference Paper

JOSAL: Joint Learning Framework for Open-Set Active Learning

Jun Xie
Xiaohui Song
Yangjie Cao
Zhi Liu 0002
Weiping Wang
Hongli Xu

Previous research in active learning has primarily focused on selecting examples from closed-set data, which consists solely of unlabeled examples from the target classes. However, this approach overlooks the more prevalent scenario of open-set data in real-world applications. Open-set data encompasses examples from both target classes and non-target classes. To fill this gap, we propose a novel framework called JOSAL, which enhances the accuracy of the classifier by precisely selecting the target class examples from open-set data. The JOSAL framework introduces the concept of joint learning, where the Sampler and Classifier components perform sampling and classification tasks, respectively, by sharing example features extracted from a pre-trained Encoder. To maximize the classification accuracy of the Classifier, the framework adopts a novel joint learning strategy. This strategy initially prioritizes optimizing the Sampler and gradually shifts the optimization attention to the Classifier. The experimental results demonstrate that, compared to baselines, our approach exhibits stronger sampling precision and achieves higher classification accuracy. To the best of our knowledge, this is the first work to address the open-set active learning problem using the joint learning paradigm.

Details

NeurIPS Conference 2023 Conference Paper

EMMA-X: An EM-like Multilingual Pre-training Algorithm for Cross-lingual Representation Learning

Ping Guo
Xiangpeng Wei
Yue Hu
Baosong Yang
Dayiheng Liu
Fei Huang
Jun Xie

Expressing universal semantics common to all languages is helpful to understand the meanings of complex and culture-specific sentences. The research theme underlying this scenario focuses on learning universal representations across languages with the usage of massive parallel corpora. However, due to the sparsity and scarcity of parallel data, there is still a big challenge in learning authentic ``universals'' for any two languages. In this paper, we propose Emma-X: an EM-like Multilingual pre-training Algorithm, to learn Cross-lingual universals with the aid of excessive multilingual non-parallel data. Emma-X unifies the cross-lingual representation learning task and an extra semantic relation prediction task within an EM framework. Both the extra semantic classifier and the cross-lingual sentence encoder approximate the semantic relation of two sentences, and supervise each other until convergence. To evaluate Emma-X, we conduct experiments on xrete, a newly introduced benchmark containing 12 widely studied cross-lingual tasks that fully depend on sentence-level representations. Results reveal that Emma-X achieves state-of-the-art performance. Further geometric analysis of the built representation space with three requirements demonstrates the superiority of Emma-X over advanced models.

PDF Details

ICRA Conference 2023 Conference Paper

Natural Language Instruction Understanding for Robotic Manipulation: a Multisensory Perception Approach

Weihua Wang
Xiaofei Li
Yanzhi Dong
Jun Xie
Di Guo 0002
Huaping Liu 0001

It has always been expected that the robot can understand the natural language instruction and thus a more natural human-robot interaction is achieved. Currently, the robot usually interprets the instruction by visually grounding the textual information to its surroundings, while it may be not enough for some complex situations with only visual perception. So it is reasonable for the robot to leverage its multisensory perception ability to better understand the instruction. In this paper, we propose a multisensory perception approach to tackle the task of natural language instruction understanding for robotic manipulation, in which the robot coordinates its visual, tactile and auditory perception to fully understand the instruction and then executes the manipulation task. Extensive experiments have been conducted demonstrating the superiority of the multisensory perception compared with single sensory perception for instruction understanding. Moreover, we establish a user-friendly human-robot interaction interface where the human sends instruction to the robot via a mobile APP.

Details

AAAI Conference 2022 Conference Paper

Non-parametric Online Learning from Human Feedback for Neural Machine Translation

Dongqi Wang
Haoran Wei
Zhirui Zhang
Shujian Huang
Jun Xie
Jiajun Chen

We study the problem of online learning with human feedback in the human-in-the-loop machine translation, in which the human translators revise the machine-generated translations and then the corrected translations are used to improve the neural machine translation (NMT) system. However, previous methods require online model updating or additional translation memory networks to achieve high-quality performance, making them inflexible and inefficient in practice. In this paper, we propose a novel non-parametric online learning method without changing the model structure. This approach introduces two k-nearest-neighbor (KNN) modules: one module memorizes the human feedback, which is the correct sentences provided by human translators, while the other balances the usage of the history human feedback and original NMT models adaptively. Experiments conducted on EMEA and JRC-Acquis benchmarks demonstrate that our proposed method obtains substantial improvements on translation accuracy and achieves better adaptation performance with less repeating human correction operations.

PDF Details

AAAI Conference 2022 Conference Paper

Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement

Yichao Du
Zhirui Zhang
Weizhi Wang
Boxing Chen
Jun Xie
Tong Xu

End-to-end speech-to-text translation (E2E-ST) is becoming increasingly popular due to the potential of its less error propagation, lower latency, and fewer parameters. Given the triplet training corpus ⟨speech, transcription, translation⟩, the conventional high-quality E2E-ST system leverages the ⟨speech, transcription⟩ pair to pre-train the model and then utilizes the ⟨speech, translation⟩ pair to optimize it further. However, this process only involves two-tuple data at each stage, and this loose coupling fails to fully exploit the association between triplet data. In this paper, we attempt to model the joint probability of transcription and translation based on the speech input to directly leverage such triplet data. Based on that, we propose a novel regularization method for model training to improve the agreement of dual-path decomposition within triplet data, which should be equal in theory. To achieve this goal, we introduce two Kullback-Leibler divergence regularization terms into the model training objective to reduce the mismatch between output probabilities of dual-path. Then the well-trained model can be naturally transformed as the E2E-ST models by the pre-defined early stop tag. Experiments on the MuST-C benchmark demonstrate that our proposed approach significantly outperforms state-of-theart E2E-ST baselines on all 8 language pairs, while achieving better performance in the automatic speech recognition task.

PDF Details

NeurIPS Conference 2021 Conference Paper

STEP: Segmenting and Tracking Every Pixel

Mark Weber
Jun Xie
Yukun Zhu
Paul Voigtlaender
Bo Chen
Bradley Green
Andreas Geiger
Bastian Leibe

The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-world setting requiring dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. To overcome this, we introduce a new benchmark encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP. The datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking under real-world conditions. We further propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is more appropriate for evaluating sequences of arbitrary length. Finally, we provide several baselines to evaluate the status of existing methods on this new challenging dataset. We have made our datasets, metric, benchmark servers, and baselines publicly available, and hope this will inspire future research.

PDF Details

AAAI Conference 2020 Conference Paper

Neural Simile Recognition with Cyclic Multitask Learning and Local Attention

Jiali Zeng
Linfeng Song
Jinsong Su
Jun Xie
Wei Song
Jiebo Luo

Simile recognition is to detect simile sentences and to extract simile components, i. e. , tenors and vehicles. It involves two subtasks: simile sentence classiﬁcation and simile component extraction. Recent work has shown that standard multitask learning is effective for Chinese simile recognition, but it is still uncertain whether the mutual effects between the subtasks have been well captured by simple parameter sharing. We propose a novel cyclic multitask learning framework for neural simile recognition, which stacks the subtasks and makes them into a loop by connecting the last to the ﬁrst. It iteratively performs each subtask, taking the outputs of the previous subtask as additional inputs to the current one, so that the interdependence between the subtasks can be better explored. Extensive experiments show that our framework signiﬁcantly outperforms the current state-of-the-art model and our carefully designed baselines, and the gains are still remarkable using BERT. Source Code of this paper are available on https: //github. com/DeepLearnXMU/Cyclic.

PDF Details

AAAI Conference 2018 Conference Paper

Deep Semantic Role Labeling With Self-Attention

Zhixing Tan
Mingxuan Wang
Jun Xie
Yidong Chen
Xiaodong Shi

Semantic Role Labeling (SRL) is believed to be a crucial step towards natural language understanding and has been widely studied. Recent years, end-to-end SRL with recurrent neural networks (RNN) has gained increasing attention. However, it remains a major challenge for RNNs to handle structural information and long range dependencies. In this paper, we present a simple and effective architecture for SRL which aims to address these problems. Our model is based on self-attention which can directly capture the relationships between two tokens regardless of their distance. Our single model achieves F1 = 83. 4 on the CoNLL-2005 shared task dataset and F1 = 82. 7 on the CoNLL-2012 shared task dataset, which outperforms the previous state-of-the-art results by 1. 8 and 1. 0 F1 score respectively. Besides, our model is computationally efﬁcient, and the parsing speed is 50K tokens per second on a single Titan X GPU.

PDF Details

AAAI Conference 2015 Conference Paper

Learning Greedy Policies for the Easy-First Framework

Jun Xie
Chao Ma
Janardhan Rao Doppa
Prashanth Mannem
Xiaoli Fern
Thomas G. Dietterich
Prasad Tadepalli

Easy-first, a search-based structured prediction approach, has been applied to many NLP tasks including dependency parsing and coreference resolution. This approach employs a learned greedy policy (action scoring function) to make easy decisions first, which constrains the remaining decisions and makes them easier. We formulate greedy policy learning in the Easy-first approach as a novel non-convex optimization problem and solve it via an efficient Majorization Minimization (MM) algorithm. Results on within-document coreference and cross-document joint entity and event coreference tasks demonstrate that the proposed approach achieves statistically significant performance improvement over existing training regimes for Easy-first and is less susceptible to overfitting.

PDF Details