Arrow Research search

Author name cluster

Yi Zhu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

33 papers
2 author rows

Possible papers

33

AIJ Journal 2026 Journal Article

Analyzing bullet chats for recommendation intent identification: Dataset and method

  • Yi Zhu
  • Qinqin Han
  • Yunhao Yuan
  • Chaowei Zhang
  • Jipeng Qiang
  • Xindong Wu

Live-streaming sales (LS) have emerged as a major e-commerce model particularly in China, with bullet chats serving as the primary channel for audience interaction. Bullet chats refer to real-time scrolling comments, these short, fast-flowing messages create unique analytical challenges, particularly for sellers who must identify and respond to key messages promptly. In this paper, we present the first investigation on bullet chats for live-streaming sales, with a focus on the critical task of Recommendation Intent Identification (RII), i. e. , determining whether a bullet chat reflects a recommendation-seeking intent or is merely casual commentary. Specifically, we construct and release the first publicly available benchmark dataset for RII, termed BC4RII, which comprises 0. 14 million bullet chats collected from four mainstream LS platforms. Furthermore, we propose SPT-RII, an advanced method on the ground of soft prompt-tuning tailored for the RII task. SPT-RII enriches sparse bullet-chat semantics through explanation generation and adapts to streaming input via a dynamic vocabulary update mechanism. Experimental results demonstrate that our method significantly outperforms state-of-the-art baselines, including large language models. To the best of our knowledge, this is the first comprehensive study analyzing bullet chats for RII on live-streaming sales, which establishes a foundational resource and method for real-time intent understanding in live-streaming sales.

Details DOI

EAAI Journal 2026 Journal Article

Causal-stabilized latent regression for robust industrial soft sensors

  • Kepeng Qiu
  • Baowei Rong
  • Yi Zhu
  • Yu Liu
  • Hui Yan
  • Haijun Cao
  • Weiwei Wang

Soft sensors play an essential role in monitoring and optimizing industrial processes. However, data redundancy and changing operating conditions often limit their reliability and ability to work well in different situations. This paper presents causal-stabilized latent regression (CSLR), a novel framework that combines causal inference with adaptive weighting. This combination helps balance feature importance and makes the model more stable when conditions change. The framework has two main parts working together: First, the causal balance weight optimization module reduces redundancy and aligns data distributions by optimizing both sample and feature weights through a causal weighting approach. Second, the robust weighted partial least squares (PLS) modeling module uses these weights to build a regression model that focuses on causally important features while reducing multicollinearity problems. Our analysis shows that CSLR effectively reduces feature redundancy and improves both generalization and stability when process conditions change. Tests on industrial debutanizer and fermentation processes demonstrate that CSLR achieves significant improvements over existing methods, with prediction error reduced by up to 43. 5%, R 2 increased by 8. 88%, and mean absolute percentage error decreased by 60. 62%, confirming its effectiveness for building accurate and reliable soft sensors.

Details DOI

EAAI Journal 2026 Journal Article

Discovering constitutive models for clay through physics-guided symbolic regression

  • Su Chen
  • Yi Zhu
  • Suyang Wang
  • Guosheng Wang
  • Xiaojun Li

The mechanical behavior of geotechnical materials is mathematically characterized through constitutive models, which are typically formulated based on a priori assumptions and theoretical deductions to effectively address a wide range of engineering challenges. Given the inherent complexity of soil, a trade-off exists between the accuracy of these models and their practical applicability in engineering applications. This study introduces a modeling framework that incorporates physical constraints into symbolic regression (SR). By utilizing finite mathematical operators and variables, the framework guides the optimization process from both physical and mechanical perspectives, yielding a concise yet robust description of constitutive models. Comparative evaluations against purely data-driven models and classical physics-based models indicate that the integration of physical and mechanical constraints significantly improves the efficiency and effectiveness of model optimization. The proposed approach achieves high predictive accuracy while maintaining low model complexity, without requiring high-resolution input data. This hybrid modeling framework effectively balances physical discovery with data fitting, demonstrating robust generalization and extrapolation capabilities.

Details DOI

EAAI Journal 2026 Journal Article

LLM4CGDS: Large language model-based agents for Chinese graded document simplification

  • Dengzhao Fang
  • Jipeng Qiang
  • Wenjie Hou
  • Yi Zhu
  • Jingtong Gao
  • Xiangyu Zhao

Graded reading tailors text difficulty to learners’ proficiency by producing multiple versions of the same content—an approach long embraced in language education but still dependent on labor-intensive, expert-driven adaptation. In this paper, we introduce the task of Chinese Graded Document Simplification (CGDS) for non-native learners, which seeks to automate the creation of multi-level reading materials in accordance with established proficiency standards. Guided by the three stages of the Hanyu Shuiping Kaoshi (HSK) 3. 0 framework (Levels 1–3 for Advanced, Levels 4–6 for Intermediate, and Levels 7–9 for Beginner learners), we propose Large Language Model for Chinese Graded Document Simplification (LLM4CGDS), a rule-guided, large language model (LLM)-based framework that integrates HSK-level readability constraints and external knowledge retrieval to control document-level simplification without requiring supervised fine-tuning. To foster further research, we construct two complementary datasets: Journey to the West Document Simplification (JWDS) and Multi-Domain Document Simplification (MDDS) that covering diverse genres and difficulty levels. Experimental evaluation on two datasets demonstrates that LLM4CGDS substantially outperforms direct prompting of state-of-the-art LLMs in both readability control and meaning preservation.

Details DOI

EAAI Journal 2026 Journal Article

SubAttack: A word-level adversarial textual attack method via antonym substitution

  • Chenqi Hua
  • Xiaojian Liu
  • Yi Zhu
  • Chaowei Zhang
  • Yun Li
  • Yunhao Yuan
  • Jipeng Qiang

Over the past few years, various word-level textual attack approaches have been proposed to reveal the vulnerability in existing deep neural networks and even large language models (LLMs) for Natural Language Processing (NLP). The textual attack aims to fool existing models into making erroneous predictions by altering the text without affecting the user’s understanding. However, current methods either struggle to construct semantically preserved adversarial texts and altered the semantics of the original text, or fail to consider the semantic perturbation constraints and are prone to invalid adversarial examples. In this paper, we propose an efficient and effective framework SubAttack to address these issues. SubAttack is a word-level adversarial textual attack method via antonym substitution, which replaces semantic indicator keywords to generate high-quality adversarial samples with considering both semantically preservation and semantic perturbation. Specifically, the process first involves tokenizing the text and performing part-of-speech tagging Identifying the semantic indicator keywords. Then, the antonym ranking is designed to decide the substitutions of candidate words to fit the context. Finally, while retaining the original text, the ranked antonyms are integrated into the text and the instructions are added for both semantically preservation and semantic perturbation. Extensive experiments reveal that state-of-the-art (SOTA) LLMs (e. g. Llama and QWen) are still vulnerable to our SubAttack. Further experiments show that the adversarial examples crafted by SubAttack usually have higher quality, exhibit better fluency and barely affect human performance and can bring more robustness improvement to victim models by adversarial training.

Details DOI

EAAI Journal 2025 Journal Article

A domain adaptation method to Defend Chinese textual adversarial attacks via prompt-tuning

  • Yi Zhu
  • Zhenglong Li
  • Yun Li
  • Yunhao Yuan
  • Jipeng Qiang

The textual adversarial attack aims to fool existing models into making erroneous predictions by adding strategic perturbations to normal data without affecting the user’s understanding. Recently, methods based on Pre-trained Language Models (PLMs) and Large Language Models (LLMs) have shown promising performance in various Natural Language Processing (NLP) downstream tasks. However, due to significant deviations between the original and perturbed texts, these methods struggle to achieve satisfactory results in defending against textual adversarial attacks, especially in Chinese, which has unique syntactic structures. To address this issue, we propose a domain adaptation method for defending against Chinese textual adversarial attacks through a prompt-tuning model, which effectively mitigates the discrepancy between different domains. Specifically, the original and perturbed texts are treated as the source and target domains, respectively, with the textual adversarial defense task framed as a cross-domain classification problem. The soft prompt-tuning model trained in the source domain is iteratively adapted to uncover the true label information in the target domain. The graph attention network is incorporated to integrate Chinese syntactic structure information with semantic features. Through a voting mechanism on predicted labels generated by the iterative model, soft prompt-tuning is further optimized for cross-domain classification tasks. Extensive experimental results demonstrate the superior effectiveness of our method in Chinese textual adversarial defense tasks compared to baseline methods, including the state-of-the-art fine-tuning approaches for PLMs and LLMs.

Details DOI

ICLR Conference 2025 Conference Paper

Differential Transformer

  • Tianzhu Ye
  • Li Dong 0004
  • Yuqing Xia
  • Yutao Sun
  • Yi Zhu
  • Gao Huang 0001
  • Furu Wei

Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax attention maps. The subtraction cancels noise, promoting the emergence of sparse attention patterns. Experimental results on language modeling show that Diff Transformer outperforms Transformer in various settings of scaling up model size and training tokens. More intriguingly, it offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. By being less distracted by irrelevant context, Diff Transformer can mitigate hallucination in question answering and text summarization. For in-context learning, Diff Transformer not only enhances accuracy but is also more robust to order permutation, which was considered as a chronic robustness issue. The results position Diff Transformer as a highly effective and promising architecture for large language models.

Details

TMLR Journal 2025 Journal Article

Potential Score Matching: Debiasing Molecular Structure Sampling with Potential Energy Guidance

  • Liya Guo
  • Zun Wang
  • Chang Liu
  • Junzhe Li
  • Pipi Hu
  • Yi Zhu
  • Tao Qin

The ensemble average of physical properties of molecules is closely related to the distribution of molecular conformations, and sampling such distributions is a fundamental challenge in physics and chemistry. Traditional methods like molecular dynamics (MD) simulations and Markov chain Monte Carlo (MCMC) sampling are commonly used but can be time-consuming and costly. Recently, diffusion models have emerged as efficient alternatives by learning the distribution of training data. Obtaining an unbiased target distribution is still an expensive task, primarily because it requires satisfying ergodicity. To tackle these challenges, we propose Potential Score Matching (PSM), an approach that utilizes the potential energy gradient to guide generative models. PSM does not require exact energy functions and can debias sample distributions even when trained on limited and biased data. Our method outperforms existing state-of-the-art (SOTA) models on the Lennard-Jones (LJ) potential, a commonly used toy model. Furthermore, we extend the evaluation of PSM to high-dimensional problems using the MD17 and MD22 datasets. The results demonstrate that molecular distributions generated by PSM more closely approximate the Boltzmann distribution compared to traditional diffusion models.

PDF Details

NeurIPS Conference 2025 Conference Paper

rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset

  • Yifei Liu
  • Li Lyna Zhang
  • Yi Zhu
  • Bingcheng Dong
  • Xudong Zhou
  • Ning Shang
  • Fan Yang
  • Cheng Li

Advancing code reasoning in large language models (LLMs) is fundamentally limited by the scarcity of high-difficulty datasets, especially those with verifiable input-output test cases necessary for rigorous solution validation at scale. We introduce rStar-Coder, which significantly improves LLM code reasoning capabilities by constructing a large-scale, verified dataset of 418K competition-level code problems, 580K long-reasoning solutions along with rich test cases of varying difficulty. This is achieved through three core contributions: (1) we curate competitive programming code problems and solutions to synthesize new, solvable problems; (2) we introduce a reliable input-output test case synthesis pipeline that decouples the generation into a three-step input generation method and a mutual verification mechanism for effective output labeling; (3) we augment problems with high-quality, test-case-verified long-reasoning solutions. Extensive experiments on Qwen models (1. 5B-14B) across various code reasoning benchmarks demonstrate the superiority of rStar-Coder dataset, achieving leading performance comparable to frontier reasoning LLMs with significantly smaller model sizes. On LiveCodeBench, rStar-Coder improves Qwen2. 5-7B from 17. 4% to an impressive 57. 3%, and Qwen2. 5-14B from 23. 3% to 62. 5%, surpassing o3-mini (low) by 3. 1%. On the more challenging USA Computing Olympiad, our 7B model achieves an average pass@1 accuracy of 16. 15%, outperforming the frontier-level QWQ-32B. rStar-Coder dataset is publicly available at https: //huggingface. co/datasets/microsoft/rStar-Coder.

PDF Details

ICML Conference 2025 Conference Paper

rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

  • Xinyu Guan
  • Li Lyna Zhang
  • Yifei Liu
  • Ning Shang
  • Youran Sun
  • Yi Zhu
  • Fan Yang 0024
  • Mao Yang 0004

We present rStar-Math to demonstrate that small language models (SLMs) can rival or even surpass the math reasoning capability of OpenAI o1, without distillation from superior models. rStar-Math achieves this by exercising “deep thinking” through Monte Carlo Tree Search (MCTS), where a math policy SLM performs test-time search guided by an SLM-based process reward model. rStar-Math introduces three innovations to tackle the challenges in training the two SLMs: (1) a novel code-augmented CoT data synthesis method, which performs extensive MCTS rollouts to generate step-by-step verified reasoning trajectories used to train the policy SLM; (2) a novel process reward model training method that avoids naïve step-level score annotation, yielding a more effective process preference model (PPM); (3) a self-evolution recipe in which the policy SLM and PPM are built from scratch and iteratively evolved to improve reasoning capabilities. Through 4 rounds of self-evolution with millions of synthesized solutions for 747k math problems, rStar-Math boosts SLMs’ math reasoning to state-of-the-art levels. On MATH benchmark, it improves Qwen2. 5-Math-7B from 58. 8% to 90. 0%, surpassing o1-preview by +4. 5%. On the USA Math Olympiad (AIME), rStar-Math solves an average of 53. 3% (8/15) of problems, ranking among the top 20% of the brightest high school math students. Code and data are available at https: //github. com/microsoft/rStar.

Details

EAAI Journal 2025 Journal Article

Soft Prompt-tuning with Self-Resource Verbalizer for short text streams

  • Yi Zhu
  • Ye Wang
  • Yun Li
  • Jipeng Qiang
  • Yunhao Yuan

Short text streams such as real-time news and search snippets have attained vast amounts of attention and research in recent decades, the characteristics of high generation velocity, feature sparsity, and high ambiguity accentuate both the importance and challenges to language models. However, most of the existing short text stream classification methods can neither automatically select relevant knowledge components for arbitrary samples, nor expand knowledge internally instead of rely on external open knowledge base to address the inherent limitations of short text stream. In this paper, we propose a Soft Prompt-tuning with Self-Resource Verbalizer (SPSV for short) for short text stream classification, the soft prompt with self-resource knowledgeable expansion is conducted for updating label words space to address evolved semantic topics in the data streams. Specifically, the automatic constructed prompt is first generated to instruct the model prediction, which is optimized to address the problem of high velocity and topic drift in short text streams. Then, in each chunk, the projection between category names and label words space, i. e. verbalizer, is updated, which is constructed by internal knowledge expansion from the short text itself. Through comprehensive experiments on four well-known benchmark datasets, we validate the superb performance of our method compared to other short text stream classification and fine-tuning PLMs methods, which achieves up to more than 90% classification accuracy with the counts of data chunk increased.

Details DOI

JBHI Journal 2025 Journal Article

WavRx: A Disease-Agnostic, Generalizable, and Privacy-Preserving Speech Health Diagnostic Model

  • Yi Zhu
  • Tiago Falk

Speech is known to carry health-related attributes, which has emerged as a novel venue for remote and long-term health monitoring. However, existing models are usually tailored for a specific type of disease, and have been shown to lack generalizability across datasets. Furthermore, concerns have been raised recently towards the leakage of speaker identity from health embeddings. To mitigate these limitations, we propose WavRx, a speech health diagnostics model that captures the respiration and articulation related dynamics from a universal speech representation. Our in-domain and cross-domain experiments on six pathological speech datasets demonstrate WavRx as a new state-of-the-art health diagnostic model. Furthermore, we show that the amount of speaker identity entailed in the WavRx health embeddings is significantly reduced without extra guidance during training. An in-depth analysis of the model was performed, thus providing physiological interpretation of its improved generalizability and privacy-preserving ability.

Details DOI

TMLR Journal 2024 Journal Article

Efficient Large Language Models: A Survey

  • Zhongwei Wan
  • Xin Wang
  • Che Liu
  • Samiul Alam
  • Yu Zheng
  • Jiachen Liu
  • Zhongnan Qu
  • Shen Yan

Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding and language generation, and thus have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey. We will actively maintain the repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.

PDF Details

NeurIPS Conference 2024 Conference Paper

SLIM: Style-Linguistics Mismatch Model for Generalized Audio Deepfake Detection

  • Yi Zhu
  • Surya Koppisetti
  • Trang Tran
  • Gaurav Bharaj

Audio deepfake detection (ADD) is crucial to combat the misuse of speech synthesized by generative AI models. Existing ADD models suffer from generalization issues to unseen attacks, with a large performance discrepancy between in-domain and out-of-domain data. Moreover, the black-box nature of existing models limits their use in real-world scenarios, where explanations are required for model decisions. To alleviate these issues, we introduce a new ADD model that explicitly uses the Style-LInguistics Mismatch (SLIM) in fake speech to separate them from real speech. SLIM first employs self-supervised pretraining on only real samples to learn the style-linguistics dependency in the real class. The learned features are then used in complement with standard pretrained acoustic features (e. g. , Wav2vec) to learn a classifier on the real and fake classes. When the feature encoders are frozen, SLIM outperforms benchmark methods on out-of-domain datasets while achieving competitive results on in-domain data. The features learned by SLIM allow us to quantify the (mis)match between style and linguistic content in a sample, hence facilitating an explanation of the model decision.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

UNIT: Unifying Image and Text Recognition in One Vision Encoder

  • Yi Zhu
  • Yanpeng Zhou
  • Chunwei Wang
  • Yang Cao
  • Jianhua Han
  • Lu Hou
  • Hang Xu

Currently, vision encoder models like Vision Transformers (ViTs) typically excel at image recognition tasks but cannot simultaneously support text recognition like human visual recognition. To address this limitation, we propose UNIT, a novel training framework aimed at UNifying Image and Text recognition within a single model. Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces a lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities. The training process comprises two stages: intra-scale pretraining and inter-scale finetuning. During intra-scale pretraining, UNIT learns unified representations from multi-scale inputs, where images and documents are at their commonly used resolution, to enable fundamental recognition capability. In the inter-scale finetuning stage, the model introduces scale-exchanged data, featuring images and documents at resolutions different from the most commonly used ones, to enhance its scale robustness. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment. Experiments across multiple benchmarks confirm that our method significantly outperforms existing methods on document-related tasks (e. g. , OCR and DocQA) while maintaining the performances on natural images, demonstrating its ability to substantially enhance text recognition without compromising its core image recognition capabilities.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

  • Youpeng Wen
  • Junfan Lin
  • Yi Zhu
  • Jianhua Han
  • Hang Xu
  • Shen Zhao
  • Xiaodan Liang

Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests the feasibility of leveraging diverse robot trajectory data to develop a unified, dynamics-aware model to enhance robot manipulation. However, given the relatively small amount of available robot data, directly fitting data without considering the relationship between visual observations and actions could lead to suboptimal data utilization. To this end, we propose \textbf{VidMan} (\textbf{Vid}eo Diffusion for Robot \textbf{Man}ipulation), a novel framework that employs a two-stage training mechanism inspired by dual-process theory from neuroscience to enhance stability and improve data utilization efficiency. Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics. In the second stage, a flexible yet effective layer-wise self-attention adapter is introduced to transform VidMan into an efficient inverse dynamics model that predicts action modulated by the implicit dynamics knowledge via parameter sharing. Our VidMan framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11. 7\% relative improvement, and demonstrates over 9\% precision gains on the OXE small-scale dataset. These results provide compelling evidence that world models can significantly enhance the precision of robot action prediction. Codes and models will be public.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

You Only Cache Once: Decoder-Decoder Architectures for Language Models

  • Yutao Sun
  • Li Dong
  • Yi Zhu
  • Shaohan Huang
  • Wenhui Wang
  • Shuming Ma
  • Quanlu Zhang
  • Jianyong Wang

We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i. e. , a cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. We also extend YOCO to 1M context length with near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Actional Atomic-Concept Learning for Demystifying Vision-Language Navigation

  • Bingqian Lin
  • Yi Zhu
  • Xiaodan Liang
  • Liang Lin
  • Jianzhuang Liu

Vision-Language Navigation (VLN) is a challenging task which requires an agent to align complex visual observations to language instructions to reach the goal position. Most existing VLN agents directly learn to align the raw directional features and visual features trained using one-hot labels to linguistic instruction features. However, the big semantic gap among these multi-modal inputs makes the alignment difficult and therefore limits the navigation performance. In this paper, we propose Actional Atomic-Concept Learning (AACL), which maps visual observations to actional atomic concepts for facilitating the alignment. Specifically, an actional atomic concept is a natural language phrase containing an atomic action and an object, e.g., ``go up stairs''. These actional atomic concepts, which serve as the bridge between observations and instructions, can effectively mitigate the semantic gap and simplify the alignment. AACL contains three core components: 1) a concept mapping module to map the observations to the actional atomic concept representations through the VLN environment and the recently proposed Contrastive Language-Image Pretraining (CLIP) model, 2) a concept refining adapter to encourage more instruction-oriented object concept extraction by re-ranking the predicted object concepts by CLIP, and 3) an observation co-embedding module which utilizes concept representations to regularize the observation representations. Our AACL establishes new state-of-the-art results on both fine-grained (R2R) and high-level (REVERIE and R2R-Last) VLN benchmarks. Moreover, the visualization shows that AACL significantly improves the interpretability in action decision. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/VLN-AACL.

PDF Details DOI

AIJ Journal 2023 Journal Article

Natural language watermarking via paraphraser-based lexical substitution

  • Jipeng Qiang
  • Shiyu Zhu
  • Yun Li
  • Yi Zhu
  • Yunhao Yuan
  • Xindong Wu

Although powerful pretrained language models generate high-quality output text, they bring new concerns about the potential misuse of such models for malicious purposes. Natural language watermarking (NLW) is a technique that is desgined to help tracing the provenance of texts for againsting possible attacks, where the watermark signals are embedded into cover texts using synonym substitutions. The up-to-date BERT-based NLW methods have made remarkable progress on performance improvement of watermarking through generating substitutes for a masked target word. Yet, the BERT-based NLWs focus on the context of texts rather than the meaning of target words, which might make the capacity of watermark embeddings being lower. To address the limitations, this study proposes a novel NLW method by incorporating a paraphraser-based lexical substitution method. Under the promise of paraphrase preservation, the proposed NLW method utilizes the knowledge of paraphrase modeling to generate the substitute candidates to replace the words in original sentences capable of carrying the watermark signal in local contexts. We empirically show that our NLW method not only has a better meaning-preserved, but improves the payload more than 2 times compared with the BERT-based NLW method. Besides, compared with previous state-of-the-art method Compared with other state-of-the-art baselines, the experimental results show that the proposed LS method improves the Precision@1 score from 51. 7% to 58. 3% and from 50. 5% to 62. 6% on LS07 and CoInCo benchmarks, respectively.

Details DOI

NeurIPS Conference 2023 Conference Paper

PreDiff: Precipitation Nowcasting with Latent Diffusion Models

  • Zhihan Gao
  • Xingjian Shi
  • Boran Han
  • Hao Wang
  • Xiaoyong Jin
  • Danielle Maddix
  • Yi Zhu
  • Mu Li

Earth system forecasting has traditionally relied on complex physical models that are computationally expensive and require significant domain expertise. In the past decade, the unprecedented increase in spatiotemporal Earth observation data has enabled data-driven forecasting models using deep learning techniques. These models have shown promise for diverse Earth system forecasting tasks but either struggle with handling uncertainty or neglect domain-specific prior knowledge, resulting in averaging possible futures to blurred forecasts or generating physically implausible predictions. To address these limitations, we propose a two-stage pipeline for probabilistic spatiotemporal forecasting: 1) We develop PreDiff, a conditional latent diffusion model capable of probabilistic forecasts. 2) We incorporate an explicit knowledge alignment mechanism to align forecasts with domain-specific physical constraints. This is achieved by estimating the deviation from imposed constraints at each denoising step and adjusting the transition distribution accordingly. We conduct empirical studies on two datasets: N-body MNIST, a synthetic dataset with chaotic behavior, and SEVIR, a real-world precipitation nowcasting dataset. Specifically, we impose the law of conservation of energy in N-body MNIST and anticipated precipitation intensity in SEVIR. Experiments demonstrate the effectiveness of PreDiff in handling uncertainty, incorporating domain-specific prior knowledge, and generating forecasts that exhibit high operational utility.

PDF Details

NeurIPS Conference 2023 Conference Paper

Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition

  • Shuhuai Ren
  • Aston Zhang
  • Yi Zhu
  • Shuai Zhang
  • Shuai Zheng
  • Mu Li
  • Alexander J. Smola
  • Xu Sun

This work proposes POMP, a prompt pre-training method for vision-language models. Being memory and computation efficient, POMP enables the learned prompt to condense semantic information for a rich set of visual concepts with over twenty-thousand classes. Once pre-trained, the prompt with a strong transferable ability can be directly plugged into a variety of visual recognition tasks including image classification, semantic segmentation, and object detection, to boost recognition performances in a zero-shot manner. Empirical evaluation shows that POMP achieves state-of-the-art performances on 21 datasets, e. g. , 67. 0% average accuracy on 10 classification datasets (+3. 1% compared to CoOp) and 84. 4 hIoU on open-vocabulary Pascal VOC segmentation (+6. 9 compared to ZSSeg).

PDF Details

ICLR Conference 2023 Conference Paper

Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations

  • Andrii Zadaianchuk
  • Matthäus Kleindessner
  • Yi Zhu
  • Francesco Locatello
  • Thomas Brox

In this paper, we show that recent advances in self-supervised representation learning enable unsupervised object discovery and semantic segmentation with a performance that matches the state of the field on supervised semantic segmentation 10 years ago. We propose a methodology based on unsupervised saliency masks and self-supervised feature clustering to kickstart object discovery followed by training a semantic segmentation network on pseudo-labels to bootstrap the system on images with multiple objects. We show that while being conceptually simple our proposed baseline is surprisingly strong. We present results on PASCAL VOC that go far beyond the current state of the art (50.0 mIoU), and we report for the first time results on MS COCO for the whole set of 81 classes: our method discovers 34 categories with more than 20% IoU, while obtaining an average IoU of 19.6 for all 81 categories.

Details

AAAI Conference 2022 Conference Paper

Contrastive Instruction-Trajectory Learning for Vision-Language Navigation

  • Xiwen Liang
  • Fengda Zhu
  • Yi Zhu
  • Bingqian Lin
  • Bing Wang
  • Xiaodan Liang

The vision-language navigation (VLN) task requires an agent to reach a target with the guidance of natural language instruction. Previous works learn to navigate step-bystep following an instruction. However, these works may fail to discriminate the similarities and discrepancies across instruction-trajectory pairs and ignore the temporal continuity of sub-instructions. These problems hinder agents from learning distinctive vision-and-language representations, harming the robustness and generalizability of the navigation policy. In this paper, we propose a Contrastive Instruction-Trajectory Learning (CITL) framework that explores invariance across similar data samples and variance across different ones to learn distinctive representations for robust navigation. Specifically, we propose: (1) a coarse-grained contrastive learning objective to enhance vision-and-language representations by contrasting semantics of full trajectory observations and instructions, respectively; (2) a fine-grained contrastive learning objective to perceive instructions by leveraging the temporal information of the sub-instructions; (3) a pairwise sample-reweighting mechanism for contrastive learning to mine hard samples and hence mitigate the influence of data sampling bias in contrastive learning. Our CITL can be easily integrated with VLN backbones to form a new learning paradigm and achieve better generalizability in unseen environments. Extensive experiments show that the model with CITL surpasses the previous state-of-the-art methods on R2R, R4R, and RxR. Code is available at https: //github. com/ liangcici/CITL-VLN.

PDF Details

NeurIPS Conference 2022 Conference Paper

CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation

  • Zicheng Zhang
  • Yi Zhu
  • Jianzhuang Liu
  • Xiaodan Liang
  • Wei Ke

Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects, but ignore the semantic consistency of pixels within the same object, leading to incomplete masks and localization errors in predictions. To tackle this problem, we propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method, to couple sentence-mask alignment with word-pixel alignment to enforce object mask constraint for achieving more accurate localization and segmentation. Specifically, the Word-Pixel Alignment (WPA) module performs early fusion of linguistic and pixel-level features in intermediate layers of the vision and language encoders. Based on the word-pixel aligned embedding, a set of mask proposals are generated to hypothesize possible objects. Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target. To further enhance the learning of the two alignment modules, an auxiliary loss is designed to contrast the foreground and background pixels. By hierarchically aligning pixels and masks with linguistic features, our CoupAlign captures the pixel coherence at both visual and semantic levels, thus generating more accurate predictions. Extensive experiments on popular datasets (e. g. , RefCOCO and G-Ref) show that our method achieves consistent improvements over state-of-the-art methods, e. g. , about 2% oIoU increase on the validation and testing set of RefCOCO. Especially, CoupAlign has remarkable ability in distinguishing the target from multiple objects of the same class. Code will be available at https: //gitee. com/mindspore/models/tree/master/research/cv/CoupAlign.

PDF Details

NeurIPS Conference 2022 Conference Paper

Earthformer: Exploring Space-Time Transformers for Earth System Forecasting

  • Zhihan Gao
  • Xingjian Shi
  • Hao Wang
  • Yi Zhu
  • Yuyang (Bernie) Wang
  • Mu Li
  • Dit-Yan Yeung

Conventionally, Earth system (e. g. , weather and climate) forecasting relies on numerical simulation with complex physical models and hence is both expensive in computation and demanding on domain expertise. With the explosive growth of spatiotemporal Earth observation data in the past decade, data-driven models that apply Deep Learning (DL) are demonstrating impressive potential for various Earth system forecasting tasks. The Transformer as an emerging DL architecture, despite its broad success in other domains, has limited adoption in this area. In this paper, we propose Earthformer, a space-time Transformer for Earth system forecasting. Earthformer is based on a generic, flexible and efficient space-time attention block, named Cuboid Attention. The idea is to decompose the data into cuboids and apply cuboid-level self-attention in parallel. These cuboids are further connected with a collection of global vectors. We conduct experiments on the MovingMNIST dataset and a newly proposed chaotic $N$-body MNIST dataset to verify the effectiveness of cuboid attention and figure out the best design of Earthformer. Experiments on two real-world benchmarks about precipitation nowcasting and El Niño/Southern Oscillation (ENSO) forecasting show that Earthformer achieves state-of-the-art performance.

PDF Details

NeurIPS Conference 2021 Conference Paper

Blending Anti-Aliasing into Vision Transformer

  • Shengju Qian
  • Hao Shao
  • Yi Zhu
  • Mu Li
  • Jiaya Jia

The transformer architectures, based on self-attention mechanism and convolution-free design, recently found superior performance and booming applications in computer vision. However, the discontinuous patch-wise tokenization process implicitly introduces jagged artifacts into attention maps, arising the traditional problem of aliasing for vision transformers. Aliasing effect occurs when discrete patterns are used to produce high frequency or continuous information, resulting in the indistinguishable distortions. Recent researches have found that modern convolution networks still suffer from this phenomenon. In this work, we analyze the uncharted problem of aliasing in vision transformer and explore to incorporate anti-aliasing properties. Specifically, we propose a plug-and-play Aliasing-Reduction Module (ARM) to alleviate the aforementioned issue. We investigate the effectiveness and generalization of the proposed method across multiple tasks and various vision transformer families. This lightweight design consistently attains a clear boost over several famous structures. Furthermore, our module also improves data efficiency and robustness of vision transformers.

PDF Details

NeurIPS Conference 2021 Conference Paper

Progressive Coordinate Transforms for Monocular 3D Object Detection

  • Li Wang
  • Li Zhang
  • Yi Zhu
  • Zhi Zhang
  • Tong He
  • Mu Li
  • Xiangyang Xue

Recognizing and localizing objects in the 3D space is a crucial ability for an AI agent to perceive its surrounding environment. While significant progress has been achieved with expensive LiDAR point clouds, it poses a great challenge for 3D object detection given only a monocular image. While there exist different alternatives for tackling this problem, it is found that they are either equipped with heavy networks to fuse RGB and depth information or empirically ineffective to process millions of pseudo-LiDAR points. With in-depth examination, we realize that these limitations are rooted in inaccurate object localization. In this paper, we propose a novel and lightweight approach, dubbed {\em Progressive Coordinate Transforms} (PCT) to facilitate learning coordinate representations. Specifically, a localization boosting mechanism with confidence-aware loss is introduced to progressively refine the localization prediction. In addition, semantic image representation is also exploited to compensate for the usage of patch proposals. Despite being lightweight and simple, our strategy allows us to establish a new state-of-the-art among the monocular 3D detectors on the competitive KITTI benchmark. At the same time, our proposed PCT shows great generalization to most coordinate-based 3D detection frameworks.

PDF Details

JMLR Journal 2020 Journal Article

GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing

  • Jian Guo
  • He He
  • Tong He
  • Leonard Lausen
  • Mu Li
  • Haibin Lin
  • Xingjian Shi
  • Chenguang Wang

We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also provide modular APIs with flexible building blocks to enable efficient customization. Leveraging the MXNet ecosystem, the deep learning models in GluonCV and GluonNLP can be deployed onto a variety of platforms with different programming languages. The Apache 2.0 license has been adopted by GluonCV and GluonNLP to allow for software distribution, modification, and usage. [abs] [ pdf ][ bib ] &copy JMLR 2020. ( edit, beta )

PDF Details

AIIM Journal 2020 Journal Article

Implementation of artificial intelligence in medicine: Status analysis and development suggestions

  • Yifan Xiang
  • Lanqin Zhao
  • Zhenzhen Liu
  • Xiaohang Wu
  • Jingjing Chen
  • Erping Long
  • Duoru Lin
  • Yi Zhu

The general public’s attitudes, demands, and expectations regarding medical AI could provide guidance for the future development of medical AI to satisfy the increasing needs of doctors and patients. The objective of this study is to investigate public perceptions, receptivity, and demands regarding the implementation of medical AI. An online questionnaire was designed to investigate the perceptions, receptivity, and demands of general public regarding medical AI between October 13 and October 30, 2018. The distributions of the current achievements, public perceptions, receptivity, and demands among individuals in different lines of work (i. e. , healthcare vs non-healthcare) and different age groups were assessed by performing descriptive statistics. The factors associated with public receptivity of medical AI were assessed using a linear regression model. In total, 2, 780 participants from 22 provinces were enrolled. Healthcare workers accounted for 54. 3 % of all participants. There was no significant difference between the healthcare workers and non-healthcare workers in the high proportion (99 %) of participants expressing acceptance of AI (p = 0. 8568), but remarkable distributional differences were observed in demands (p < 0. 001 for both demands for AI assistance and the desire for AI improvements) and perceptions (p < 0. 001 for safety, validity, trust, and expectations). High levels of receptivity (approximately 100 %), demands (approximately 80 %), and expectations (100 %) were expressed among different age groups. The receptivity of medical AI among the non-healthcare workers was associated with gender, educational qualifications, and demands and perceptions of AI. There was a very large gap between current availability of and public demands for intelligence services (p < 0. 001). More than 90 % of healthcare workers expressed a willingness to devote time to learning about AI and participating in AI research. The public exhibits a high level of receptivity regarding the implementation of medical AI. To date, the achievements have been rewarding, and further advancements are required to satisfy public demands. There is a strong demand for intelligent assistance in many medical areas, including imaging and pathology departments, outpatient services, and surgery. More contributions are imperative to facilitate integrated and advantageous implementation in medical AI.

Details DOI

YNIMG Journal 2020 Journal Article

Instructor-learner brain coupling discriminates between instructional approaches and predicts learning

  • Yafeng Pan
  • Suzanne Dikker
  • Pavel Goldstein
  • Yi Zhu
  • Cuirong Yang
  • Yi Hu

The neural mechanisms that support naturalistic learning via effective pedagogical approaches remain elusive. Here we used functional near-infrared spectroscopy to measure brain activity from instructor-learner dyads simultaneously during dynamic conceptual learning. Results revealed that brain-to-brain coupling was correlated with learning outcomes, and, crucially, appeared to be driven by specific scaffolding behaviors on the part of the instructors (e. g. , asking guiding questions or providing hints). Brain-to-brain coupling enhancement was absent when instructors used an explanation approach (e. g. , providing definitions or clarifications). Finally, we found that machine-learning techniques were more successful when decoding instructional approaches (scaffolding vs. explanation) from brain-to-brain coupling data than when using a single-brain method. These findings suggest that brain-to-brain coupling as a pedagogically relevant measure tracks the naturalistic instructional process during instructor-learner interaction throughout constructive engagement, but not information clarification.

Details DOI

AAAI Conference 2020 Conference Paper

Lexical Simplification with Pretrained Encoders

  • Jipeng Qiang
  • Yun Li
  • Yi Zhu
  • Yunhao Yuan
  • Xindong Wu

Lexical simplification (LS) aims to replace complex words in a given sentence with their simpler alternatives of equivalent meaning. Recently unsupervised lexical simplification approaches only rely on the complex word itself regardless of the given sentence to generate candidate substitutions, which will inevitably produce a large number of spurious candidates. We present a simple LS approach that makes use of the Bidirectional Encoder Representations from Transformers (BERT) which can consider both the given sentence and the complex word during generating candidate substitutions for the complex word. Specifically, we mask the complex word of the original sentence for feeding into the BERT to predict the masked token. The predicted results will be used as candidate substitutions. Despite being entirely unsupervised, experimental results show that our approach obtains obvious improvement compared with these baselines leveraging linguistic databases and parallel corpus, outperforming the state-of-the-art by more than 12 Accuracy points on three well-known benchmarks.

PDF Details

AAAI Conference 2019 Conference Paper

Tensor Decomposition for Multilayer Networks Clustering

  • Zitai Chen
  • Chuan Chen
  • Zibin Zheng
  • Yi Zhu

Clustering on multilayer networks has been shown to be a promising approach to enhance the accuracy. Various multilayer networks clustering algorithms assume all networks derive from a latent clustering structure, and jointly learn the compatible and complementary information from different networks to excavate one shared underlying structure. However, such an assumption is in conflict with many emerging real-life applications due to the existence of noisy/irrelevant networks. To address this issue, we propose Centroid-based Multilayer Network Clustering (CMNC), a novel approach which can divide irrelevant relationships into different network groups and uncover the cluster structure in each group simultaneously. The multilayer networks is represented within a unified tensor framework for simultaneously capturing multiple types of relationships between a set of entities. By imposing the rank-(Lr, Lr, 1) block term decomposition with nonnegativity, we are able to have well interpretations on the multiple clustering results based on graph cut theory. Numerically, we transform this tensor decomposition problem to an unconstrained optimization, thus can solve it efficiently under the nonlinear least squares (NLS) framework. Extensive experimental results on synthetic and real-world datasets show the effectiveness and robustness of our method against noise and irrelevant data.

PDF Details

ICRA Conference 1996 Conference Paper

A cheap output feedback tracking controller with robustness: the RLFJ problem

  • Yi Zhu
  • Darren M. Dawson
  • Timothy C. Burg
  • Jun Hu 0017

In this paper, we propose a robust output feedback link position tracking controller for the full-order model used to describe rigid-link flexible-joint (RLFJ) robot manipulators. The method uses a bank of dynamic filters, with the link position tracking error as the input, to generate surrogates for the immeasurable system states. One of these generated filter states is then used in the feedback control strategy. The method is distinguished from previous results by the limited amount of modeling information required, the utilization of only link position measurements, and the overall simplicity of the controller. Even though the approach only requires link position measurements, the controller exhibits good robustness properties while ensuring semi-global uniformly ultimately bounded link position tracking.

Details
v2026.09.13