Author name cluster

Ziyi Yang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

18 papers

2 author rows

NeurIPS Conference 2025 Conference Paper

CPSea: Large-scale cyclic peptide-protein complex dataset for machine learning in cyclic peptide design

Ziyi Yang
Hanyuan Xie
Yinjun Jia
Xiangzhe Kong
Jiqing Zheng
Ziting Zhang
Yang Liu
Lei Liu

Cyclic peptides exhibit better binding affinity and proteolytic stability compared to their linear counterparts. However, the development of cyclic peptide design models is hindered by the scarcity of data. To address this, we introduce **CPSea**(**C**yclic **P**eptide **Sea**), a dataset of 2. 71 million cyclic peptide-receptor complexes, curated through systematic mining of the AlphaFold Database (AFDB). Our pipeline extracts compact domains from AFDB, identifies cyclization sites using the $\beta$-carbon (C$_\beta$) distance thresholds, and applies multi-stage filtering to ensure structure fidelity and binding compatibility. Compared with experimental data of cyclic peptides, CPSea shows similar distributions in metrics on structure fidelity and wet-lab compatibility. To our knowledge, CPSea is the largest cyclic peptide-receptor dataset to date, enabling end-to-end model training for the first time. The dataset also showcases the feasibility of simulating inter-chain interactions using intra-chain interactions, expanding available resources for machine-learning models on protein-protein interactions. The dataset and relevant scripts are accessible on GitHub ([https: //github. com/YZY010418/CPSea](https: //github. com/YZY010418/CPSea)).

PDF Details

IJCAI Conference 2025 Conference Paper

Detecting Hallucination in Large Language Models Through Deep Internal Representation Analysis

Luan Zhang
Dandan Song
Zhijing Wu
Yuhang Tian
Changzhi Zhou
Jing Xu
Ziyi Yang
Shuhao Zhang

Large language models (LLMs) have shown exceptional performance across various domains. However, LLMs are prone to hallucinate facts and generate non-factual responses, which can undermine their reliability in real-world applications. Current hallucination detection methods suffer from external resource demands, substantial time overhead, difficulty overcoming LLMs' intrinsic limitation, and insufficient modeling. In this paper, we propose MHAD, a novel internal-representation-based hallucination detection method. MHAD utilizes linear probing to select neurons and layers within LLMs. The selected neurons and layers are demonstrated with significant awareness of hallucinations at the initial and final generation steps. By concatenating the outputs from these selected neurons of selected layers at the initial and final generation steps, a hallucination awareness vector is formed, enabling precise hallucination detection via an MLP. Additionally, we introduce SOQHD, a novel benchmark for evaluating hallucination detection in Open-Domain QA (ODQA). Extensive experiments show that MHAD outperforms existing hallucination detection methods across multiple LLMs, demonstrating superior effectiveness.

PDF Details DOI

IROS Conference 2025 Conference Paper

Least Commitment Planning for the Object Scouting Problem

Max Merlin
Ziyi Yang
George Konidaris 0001
David Paulius

State uncertainty is a primary obstacle to effective long-horizon robot task planning. State uncertainty can be decomposed into spatial uncertainty—resolved using SLAM—and uncertainty about the objects in the environment, formalized as the object scouting problem and modeled using the Locally Observable Markov Decision Process (LOMDP). We introduce a new planning framework specifically designed for object scouting with LOMDPs called the Scouting Partial-Order Planner (SPOP), which exploits the characteristics of partial order and regression planning to plan around knowledge gaps the robot may have about the existence, location, and state of relevant objects in its environment. Our results highlight the benefits of partial-order planning, demonstrating its suitability for object scouting due to its ability to identify absent but task-relevant objects, and show that it outperforms comparable planners in plan length, computation time, and execution time.

Details

TMLR Journal 2025 Journal Article

Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

Yueqin Yin
Shentao Yang
Yujia Xie
Ziyi Yang
Yuting Sun
Hany Hassan Awadalla
Weizhu Chen
Mingyuan Zhou

Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Previous RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature of LM generation and can suffer from the sparse reward issue. While recent works propose dense token-level RLHF, treating each token as an action may be oversubtle to proper reward assignment. In this paper, we seek to get the best of both by training and utilizing a segment-level reward model, which assigns a reward to each semantically complete text segment that spans over a short sequence of tokens. For reward learning, our method allows dynamic text segmentation and compatibility with standard sequence-preference datasets. For effective RL-based LM training against segment reward, we generalize the classical scalar bandit reward normalizers into location-aware normalizer functions and interpolate the segment reward for further densification. Our method performs competitively on three popular RLHF benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation studies are conducted to further demonstrate our method.

PDF Details

TMLR Journal 2025 Journal Article

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

Shenao Zhang
Donghan Yu
Hiteshi Sharma
Han Zhong
Zhihan Liu
Ziyi Yang
Shuohang Wang
Hany Hassan Awadalla

Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere to human intentions. Unlike offline alignment with a fixed dataset, online feedback collection from humans or AI on model generations typically leads to more capable reward models and better-aligned LLMs through an iterative process. However, achieving a globally accurate reward model requires systematic exploration to generate diverse responses that span the vast space of natural language. Random sampling from standard reward-maximizing LLMs alone is insufficient to fulfill this requirement. To address this issue, we propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. By solving the inner-level problem with the reparameterized reward function, the resulting algorithm, named Self-Exploring Language Models (SELM), eliminates the need for a separate RM and iteratively updates the LLM with a straightforward objective. Compared to Direct Preference Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen extrapolations and enhances exploration efficiency. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, SELM significantly boosts the performance on instruction-following benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard academic benchmarks in different settings.

PDF Details

EAAI Journal 2025 Journal Article

Super-resolution reconstruction of sequential images based on an active shift via a hybrid attention calibration mechanism

Qiang Wu
Ziyi Yang
Hongfei Zeng
Jin Zhang
Haojie Xia

Image super-resolution reconstruction converts low-resolution images into high-resolution images, demonstrating extensive potential in processing sequential images. However, most Multi-Image Super-Resolution methods currently face two significant challenges: first, the lack of precision in the shift information between images, as these methods typically rely on algorithms to estimate relative motion. Second, the limited ability to effectively extract subpixel features from low-resolution images directly impacts the richness of details in reconstructed images. This paper proposes a novel active shift-based sequential image super-resolution reconstruction technique to address these issues. This technique integrates hardware control with deep learning algorithms, utilizing a Piezoelectric platform to control camera movement precisely, capturing sequential images with predetermined subpixel shifts, and accurately recording the relative shifts between images. At the algorithmic level, we have designed a hybrid network model that combines a convolutional neural network with a Transformer architecture and integrates channel attention and self-attention mechanisms. This model fully leverages the precise shift information provided by the hardware and significantly enhances the ability to extract image details and overall image quality. Experimental results demonstrate that our method outperforms single-image super-resolution techniques regarding Peak-Signal-to-Noise-Ratio and Structural Similarity Index Measure. To further validate the applicability and effectiveness of this technology, we conducted tests using a resolution test chart, which showed that our technique can increase the resolution of the original imaging system by 25. 6%. Therefore, the strategy combining hardware and software proposed in this paper effectively solves critical issues in Multi-Image Super-Resolution tasks and provides new pathways for image processing technologies.

Details DOI

ICLR Conference 2025 Conference Paper

Weighted-Reward Preference Optimization for Implicit Model Fusion

Ziyi Yang
Fanqi Wan
Longguang Zhong
Tianyuan Shi
Xiaojun Quan

While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9\% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2\% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.

Details

AAAI Conference 2024 Conference Paper

A General Implicit Framework for Fast NeRF Composition and Rendering

Xinyu Gao
Ziyi Yang
Yunlu Zhao
Yuxiang Sun
Xiaogang Jin
Changqing Zou

A variety of Neural Radiance Fields (NeRF) methods have recently achieved remarkable success in high render speed. However, current accelerating methods are specialized and incompatible with various implicit methods, preventing real-time composition over various types of NeRF works. Because NeRF relies on sampling along rays, it is possible to provide general guidance for acceleration. To that end, we propose a general implicit pipeline for composing NeRF objects quickly. Our method enables the casting of dynamic shadows within or between objects using analytical light sources while allowing multiple NeRF objects to be seamlessly placed and rendered together with any arbitrary rigid transformations. Mainly, our work introduces a new surface representation known as Neural Depth Fields (NeDF) that quickly determines the spatial relationship between objects by allowing direct intersection computation between rays and implicit surfaces. It leverages an intersection neural network to query NeRF for acceleration instead of depending on an explicit spatial structure.Our proposed method is the first to enable both the progressive and interactive composition of NeRF objects. Additionally, it also serves as a previewing plugin for a range of existing NeRF works.

PDF Details DOI

ICRA Conference 2024 Conference Paper

Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents

Ziyi Yang
Shreyas Sundara Raman
Ankit Shah
Stefanie Tellex

Recent advancements in large language models (LLMs) have enabled a new research domain, LLM agents, for solving robotics and planning tasks by leveraging the world knowledge and general reasoning abilities of LLMs obtained during pretraining. However, while considerable effort has been made to teach the robot the "dos", the "don’ts" received relatively less attention. We argue that, for any practical usage, it is as crucial to teach the robot the "don’ts": conveying explicit instructions about prohibited actions, assessing the robot’s comprehension of these restrictions, and, most importantly, ensuring compliance. Moreover, verifiable safe operation is essential for deployments that satisfy worldwide standards such as ISO 61508, which defines standards for safely deploying robots in industrial factory environments worldwide. Aiming at deploying the LLM agents in a collaborative environment, we propose a queryable safety constraint module based on linear temporal logic (LTL) that simultaneously enables natural language (NL) to temporal constraints encoding, safety violation reasoning and explaining, and unsafe action pruning. To demonstrate the effectiveness of our system, we conducted experiments in VirtualHome environment and on a real robot. The experimental results show that our system strictly adheres to the safety constraints and scales well with complex safety constraints, highlighting its potential for practical utility.

Details

NeurIPS Conference 2024 Conference Paper

RobIR: Robust Inverse Rendering for High-Illumination Scenes

Ziyi Yang
Yanzhen Chen
Xinyu Gao
Yazhen Yuan
Yu Wu
Xiaowei Zhou
Xiaogang Jin

Implicit representation has opened up new possibilities for inverse rendering. However, existing implicit neural inverse rendering methods struggle to handle strongly illuminated scenes with significant shadows and slight reflections. The existence of shadows and reflections can lead to an inaccurate understanding of the scene, making precise factorization difficult. To this end, we present RobIR, an implicit inverse rendering approach that uses ACES tone mapping and regularized visibility estimation to reconstruct accurate BRDF of the object. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultaneously, we are able to accurately decouple environment lighting and the object's PBR materials without imposing strict constraints on the scene. Even in high-illumination scenes with shadows and specular reflections, our method can recover high-quality albedo and roughness with no shadow interference. RobIR outperforms existing methods in both quantitative and qualitative evaluations.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting

Ziyi Yang
Xinyu Gao
Yang-Tian Sun
Yi-Hua Huang
Xiaoyang Lyu
Wen Zhou
Shaohui Jiao
Xiaojuan Qi

The recent advancements in 3D Gaussian splatting (3D-GS) have not only facilitated real-time rendering through modern GPU rasterization pipelines but have also attained state-of-the-art rendering quality. Nevertheless, despite its exceptional rendering quality and performance on standard datasets, 3D-GS frequently encounters difficulties in accurately modeling specular and anisotropic components. This issue stems from the limited ability of spherical harmonics (SH) to represent high-frequency information. To overcome this challenge, we introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian. Additionally, we have developed a coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfitting in real-world scenes. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3D-GS to model scenes with specular and anisotropic components without increasing the number of 3D Gaussians. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

Any-to-Any Generation via Composable Diffusion

Zineng Tang
Ziyi Yang
Chenguang Zhu
Michael Zeng
Mohit Bansal

We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities. Unlike existing generative AI systems, CoDi can generate multiple modalities in parallel and its input is not limited to a subset of modalities like text or image. Despite the absence of training datasets for many combinations of modalities, we propose to align modalities in both the input and output space. This allows CoDi to freely condition on any input combination and generate any group of modalities, even if they are not present in the training data. CoDi employs a novel composable generation strategy which involves building a shared multimodal space by bridging alignment in the diffusion process, enabling the synchronized generation of intertwined modalities, such as temporally aligned video and audio. Highly customizable and flexible, CoDi achieves strong joint-modality generation quality, and outperforms or is on par with the unimodal state-of-the-art for single-modality synthesis.

PDF Details

ICML Conference 2023 Conference Paper

Global Selection of Contrastive Batches via Optimization on Sample Permutations

Vin Sachidananda
Ziyi Yang
Chenguang Zhu

Contrastive Learning has recently achieved state-of-the-art performance in a wide range of unimodal and multimodal tasks. Many contrastive learning approaches use mined hard negatives to make batches more informative during training but these approaches are inefficient as they increase epoch length proportional to the number of mined negatives and require frequent updates of nearest neighbor indices or mining from recent batches. In this work, we provide an alternative to hard negative mining, Global Contrastive Batch Sampling (GCBS), an efficient approximation to the batch assignment problem that upper bounds the gap between the global and training losses, $\mathcal{L}^{Global} - \mathcal{L}^{Train}$, in contrastive learning settings. Through experimentation we find GCBS improves state-of-the-art performance in sentence embedding and code-search tasks. Additionally, GCBS is easy to implement as it requires only a few additional lines of code, does not maintain external data structures such as nearest neighbor indices, is more computationally efficient than the most minimal hard negative mining approaches, and makes no changes to the model being trained. Code is available at https: //github. com/vinayak1/GCBS.

Details

AAAI Conference 2023 Conference Paper

i-Code: An Integrative and Composable Multimodal Learning Framework

Ziyi Yang
Yuwei Fang
Chenguang Zhu
Reid Pryzant
DongDong Chen
Yu Shi
Yichong Xu
Yao Qian

Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview. Most current pretraining methods, however, are limited to one or two modalities. We present i-Code, a self-supervised pretraining framework where users may flexibly combine the modalities of vision, speech, and language into unified and general-purpose vector representations. In this framework, data from each modality are first given to pretrained single-modality encoders. The encoder outputs are then integrated with a multimodal fusion network, which uses novel merge- and co-attention mechanisms to effectively combine information from the different modalities. The entire system is pretrained end-to-end with new objectives including masked modality unit modeling and cross-modality contrastive learning. Unlike previous research using only video for pretraining, the i-Code framework can dynamically process single, dual, and triple-modality data during training and inference, flexibly projecting different combinations of modalities into a single representation space. Experimental results demonstrate how i-Code can outperform state-of-the-art techniques on five multimodal understanding tasks and single-modality benchmarks, improving by as much as 11% and demonstrating the power of integrative multimodal pretraining.

PDF Details DOI

JBHI Journal 2022 Journal Article

A Home-based Tele-rehabilitation System With Enhanced Therapist-patient Remote Interaction: A Feasibility Study

Yi Liu
Shuxiang Guo
Ziyi Yang
Hideyuki Hirata
Takashi Tamiya

As a promising alternative to hospital-based manual therapy, robot-assisted tele-rehabilitation therapy has shown significant benefits in reducing the therapist's workload and accelerating the patient's recovery process. However, existing telerobotic systems for rehabilitation face barriers to implementing appropriate therapy treatment due to the lack of effective therapist-patient interactive capabilities. In this paper, we develop a home-based tele-rehabilitation system that implements two alternative training methods, including a haptic-enabled guided training that allows the therapist to adjust the intensity of therapeutic movements provided by the rehabilitation device and a surface electromyography (sEMG)-based supervised training that explores remote assessment of the patient's kinesthetic awareness. Preliminary experiments were conducted to demonstrate the feasibility of the proposed alternative training methods and evaluate the functionality of the developed tele-rehabilitation system. Results showed that the proposed tele-rehabilitation system enabled therapist-in-the-loop to dynamically adjust the rehabilitation intensity and provided more interactivity in therapist-patient remote interaction.

Details DOI

NeurIPS Conference 2022 Conference Paper

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

Zhenhailong Wang
Manling Li
Ruochen Xu
Luowei Zhou
Jie Lei
Xudong Lin
Shuohang Wang
Ziyi Yang

The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples. Existing few-shot video-language learners focus exclusively on the encoder, resulting in the absence of a video-to-text decoder to handle generative tasks. Video captioners have been pretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidIL, a few-shot Video-language Learner via Image and Language models, which demonstrates strong performance on few-shot video-to-text tasks without the necessity of pretraining or finetuning on any video datasets. We use image-language models to translate the video content into frame captions, object, attribute, and event phrases, and compose them into a temporal-aware template. We then instruct a language model, with a prompt containing a few in-context examples, to generate a target output from the composed content. The flexibility of prompting allows the model to capture any form of text input, such as automatic speech recognition (ASR) transcripts. Our experiments demonstrate the power of language models in understanding videos on a wide variety of video-language tasks, including video captioning, video question answering, video caption retrieval, and video future event prediction. Especially, on video future event prediction, our few-shot model significantly outperforms state-of-the-art supervised models trained on large-scale video datasets. Code and processed data are publicly available for research purposes at https: //github. com/MikeWangWZHL/VidIL.

PDF Details

JBHI Journal 2021 Journal Article

A Home-Based Bilateral Rehabilitation System With sEMG-based Real-Time Variable Stiffness

Yi Liu
Shuxiang Guo
Ziyi Yang
Hideyuki Hirata
Takashi Tamiya

Bilateral rehabilitation allows patients with hemiparesis to exploit the cooperative capabilities of both arms to promote the recovery process. Although various approaches have been proposed to facilitate synchronized robot-assisted bilateral movements, few studies have focused on addressing the varying joint stiffness resulting from dynamic motions. This paper presents a novel bilateral rehabilitation system that implements a surface electromyography (sEMG)-based stiffness control to achieve real-time stiffness adjustment based on the user's dynamic motion. An sEMG-driven musculoskeletal model that incorporates the muscle activation and muscular contraction dynamics is developed to provide reference signals for the robot's real-time stiffness control. Preliminary experiments were conducted to evaluate the system performance in tracking accuracy and comfortability, which showed the proposed rehabilitation system with sEMG-based real-time stiffness variation achieved fast adaption to the patient's dynamic movement as well as improving the comfort in robot-assisted bilateral training.

Details DOI

ICLR Conference 2021 Conference Paper

Filtered Inner Product Projection for Crosslingual Embedding Alignment

Vin Sachidananda
Ziyi Yang
Chenguang Zhu

Due to widespread interest in machine translation and transfer learning, there are numerous algorithms for mapping multiple embeddings to a shared representation space. Recently, these algorithms have been studied in the setting of bilingual lexicon induction where one seeks to align the embeddings of a source and a target language such that translated word pairs lie close to one another in a common representation space. In this paper, we propose a method, Filtered Inner Product Projection (FIPP), for mapping embeddings to a common representation space. As semantic shifts are pervasive across languages and domains, FIPP first identifies the common geometric structure in both embeddings and then, only on the common structure, aligns the Gram matrices of these embeddings. FIPP is applicable even when the source and target embeddings are of differing dimensionalities. Additionally, FIPP provides computational benefits in ease of implementation and is faster to compute than current approaches. Following the baselines in Glavas et al. 2019, we evaluate FIPP both in the context of bilingual lexicon induction and downstream language tasks. We show that FIPP outperforms existing methods on the XLING BLI dataset for most language pairs while also providing robust performance across downstream tasks.

Details