Author name cluster

Ziyi Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

19 papers

1 author row

YNIMG Journal 2026 Journal Article

Advancing intraoperative brain mapping: First-in-human evaluation of flexible high-density electrocorticography grids for epilepsy surgery

Sem Hoogteijling
Carmen Gelpke
Eline V. Schaft
Ziyi Wang
Tessa Souhoka
Pieter van Eijsden
Anna Diop
Karolina Janikowska

Mapping epileptiform activity, specifically high frequency oscillations (HFOs), with intraoperative electrocorticography (ECoG) may guide epilepsy surgery. The most relevant recording takes place following the surgical resection to identify any residual epileptic activity. Recording HFOs is, however, challenging due to their focal and low-amplitude nature. We designed and evaluated a novel soft, flexible, and stretchable high-density electrode grid (SOFT ECoG) tailor-made for epilepsy surgery to enhance HFO detection and enable recordings from within resection cavities. In this first-in-human evaluation of flexible high-density grids, we recorded with the SOFT ECoG grids and standard rigid high-density grids in three patients. Both grids were sequentially placed on the same cortical area and recorded similar interictal epileptiform activity (spikes and HFOs). Crucially, the SOFT ECoG grid's pliability enabled recording from relevant, but anatomically challenging, areas like the resection cavity. This was exemplified in one patient, where the SOFT ECoG identified epileptiform activity (spikes and HFOs) within a resection cavity revealing remnant epileptic tissue needing removal. This area was inaccessible to the standard rigid grid. These findings demonstrate the potential of novel flexible electrode grids to enhance intraoperative mapping of epileptiform activity including HFOs in areas inaccessible with standard rigid grids, thereby improving epilepsy surgery guidance.

Details DOI

EAAI Journal 2026 Journal Article

Cooperative decision-making of unmanned aerial vehicles: A multi-agent reinforcement learning approach

Ziyi Wang
Guoliang Ma
Jian Guo
Chen Qian
Yang Gao
Zhuo Huang

Cooperative decision-making of unmanned aerial vehicles (UAVs) for military missions is a crucial research topic. However, the ability constraints of heterogeneous UAVs in real-world scenarios bring significant challenges to the cooperative decision-making process. To address these issues, this paper proposes a multi-agent proximal policy optimization (MAPPO) algorithm with a flexible observation feature encoding (FOFE) mechanism and a Mamba-based memory structure. Firstly, the cooperative decision-making problem for reconnaissance-strike integrated fixed-wing UAV (RSUAV) swarms is formulated as a distributed partially observable Markov decision process (Dec-POMDP). Secondly, to address the variability and incompleteness in observation inputs, an FOFE strategy is introduced. This allows the network to process multi-channel and variable-length data effectively. Furthermore, the Mamba model is incorporated to capture temporal dependencies in historical observations. This enhances decision-making in prolonged missions. Under this multi-agent reinforcement learning (MARL) framework, each RSUAV can make autonomous decisions in a decentralized manner. The simulation results show that the proposed algorithm improves the completion ratio ( > 10. 1%), survival ratio ( > 14. 8%), and reduces completion time ( > 21. 5%) compared to baselines. It also exhibits strong generalization capability and holds practical feasibility for deployment on edge computing devices. Therefore, this approach enables effective cooperative decision-making under the ability constraints of heterogeneous RSUAVs.

Details DOI

AAAI Conference 2026 Conference Paper

DAWN: Distributed LLM Multi-Agent Workflow Synthesis

Guancheng Wan
Mo Zhou
Ziyi Wang
Xiaoran Shang
Eric Hanchen Jiang
Guibin Zhang
Jinhe Bi
Yunpu Ma

Large language models (LLMs) have recently empowered multi-agent systems (MAS) to achieve remarkable advances in collaborative reasoning and complex task automation. The effectiveness of these systems fundamentally depends on the design of adaptive communication graphs—the underlying workflows that coordinate agent interactions. However, in real-world scenarios, strict privacy constraints often silo data across organizations, and client distributions are highly non-IID, posing major challenges for synthesizing such workflows. In this work, we are the first to systematically study distributed multi-agent workflow synthesis under these privacy and heterogeneity constraints, and we introduce the Difficulty-Based Skew (DBS) benchmark to emulate such challenging environments. Drawing inspiration from federated graph learning (FGL)—which has primarily focused on classification over static graphs—we identify a critical gap: existing FGL methods do not address the generative design of communication topologies. We reveal two fundamental obstacles to generative workflow synthesis in this setting: (i) workflow specialization conflict, where agents optimized for different task distributions generate incompatible communication patterns that resist meaningful aggregation, and (ii) structural communication shift, where locally optimal agent interaction graphs fail to compose into globally coherent multi-agent workflows. To address these challenges, we propose DAWN, a federated framework that integrates two key innovations: Parametric Resonance, which robustly aggregates heterogeneous local updates via layer-wise SVD-based denoising and alignment, and Structural Gravity, which regularizes local workflow generation by penalizing the Fusion Gromov-Wasserstein distance to a set of prototype communication graphs, ensuring global structural coherence without stifling local adaptation. Experiments on the DBS benchmark show that DAWN surpasses baselines in global task success and reduces inter-client graph divergence, laying a solid foundation for privacy-preserving, adaptive MAS workflow design in heterogeneous settings.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Mitigating Hallucinations in Large Language Models via Causal Reasoning

Yuangang Li
Yiqing Shen
Yi Nian
Jiechao Gao
Ziyi Wang
Chenxiao Yu
Li Li
Jie Wang

Large language models (LLMs) exhibit logically inconsistent hallucinations that appear coherent yet violate reasoning principles, with recent research suggesting an inverse relationship between causal reasoning capabilities and such hallucinations. However, existing reasoning approaches in LLMs, such as Chain-of-Thought (CoT) and its graph-based variants, operate at the linguistic token level rather than modeling the underlying causal relationships between variables, lacking the ability to represent conditional independencies or satisfy causal identification assumptions. To bridge this gap, we introduce causal-DAG construction and reasoning (CDCR-SFT), a supervised fine-tuning framework that trains LLMs to explicitly construct variable-level directed acyclic graph (DAG) and then perform reasoning over it. Moreover, we present a dataset comprising 25,368 samples (CausalDR), where each sample includes an input question, explicit causal DAG, graph-based reasoning trace, and validated answer. Experiments on four LLMs across eight tasks show that CDCR-SFT improves the causal reasoning capability with the state-of-the-art 95.33% accuracy on CLADDER (surpassing human performance of 94.8% for the first time) and reduces the hallucination on HaluEval with 10% improvements. It demonstrates that explicit causal structure modeling in LLMs can effectively mitigate logical inconsistencies in LLM outputs.

PDF Details DOI

AAAI Conference 2026 Conference Paper

MP1: MeanFlow Tames Policy Learning in 1-step for Robotic Manipulation

Juyi Sheng
Ziyi Wang
Peiming Li
Mengyuan Liu

In robot manipulation, robot learning has become a prevailing approach. However, generative models within this field face a fundamental trade-off between the slow, iterative sampling of diffusion models and the architectural constraints of faster Flow-based methods, which often rely on explicit consistency losses. To address these limitations, we introduce MP1, which pairs 3D point-cloud inputs with the MeanFlow paradigm to generate action trajectories in one network function evaluation (1-NFE). By directly learning the interval-averaged velocity via the "MeanFlow Identity", our policy avoids any additional consistency constraints. This formulation eliminates numerical ODE-solver errors during inference, yielding more precise trajectories. MP1 further incorporates CFG for improved trajectory controllability while retaining 1-NFE inference without reintroducing structural constraints. Because subtle scene-context variations are critical for robot learning, especially in few-shot learning, we introduce a lightweight Dispersive Loss that repels state embeddings during training, boosting generalization without slowing inference. We validate our method on the Adroit and Meta-World benchmarks, as well as in real-world scenarios. Experimental results show MP1 achieves superior average task success rates, outperforming DP3 by 10.2% and FlowPolicy by 7.3%. Its average inference time is only 6.8 ms—19 times faster than DP3 and nearly 2 times faster than FlowPolicy.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Speech-Aware Long Context Pruning and Integration for Contextualized Automatic Speech Recognition

Yiming Rong
Yixin Zhang
Ziyi Wang
Deyang Jiang
Yunlong Zhao
Haoran Wu
Shiyu Zhou
Bo Xu

Automatic speech recognition (ASR) systems have achieved remarkable performance in common conditions but often struggle to leverage long-context information in contextualized scenarios that require domain-specific knowledge, such as conference presentations. This challenge arises primarily due to constrained model context windows and the sparsity of relevant information within extensive contextual noise. To solve this, we propose the SAP^2 method, a novel framework that dynamically prunes and integrates relevant contextual keywords in two stages. Specifically, each stage leverages our proposed Speech-Driven Attention-based Pooling mechanism, enabling efficient compression of context embeddings while preserving speech-salient information. Experimental results demonstrate state-of-the-art performance of SAP^2 on the SlideSpeech and LibriSpeech datasets, achieving word error rates (WER) of 7.71% and 1.12%, respectively. On SlideSpeech, our method notably reduces biased keyword error rates (B-WER) by 41.1% compared to non-contextual baselines. SAP^2 also exhibits robust scalability, consistently maintaining performance under extensive contextual input conditions on both datasets.

PDF Details DOI

TMLR Journal 2026 Journal Article

SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba

Yulong Huang
Jianxiong Tang
Chao Wang
Ziyi Wang
Jianguo Zhang
Zhichao Lu
Bojun Cheng
Luziwei Leng

Large Language Models (LLMs) have achieved remarkable performance across tasks but remain energy-intensive due to dense matrix operations. Spiking neural networks (SNNs) improve energy efficiency by replacing dense matrix multiplications with sparse accumulations. Their sparse spike activity enables efficient LLMs deployment on edge devices. However, prior SNN-based LLMs often sacrifice performance for efficiency, and recovering accuracy typically requires full pretraining, which is costly and impractical. To address this, we propose SpikingMamba, an energy-efficient SNN-based LLMs distilled from Mamba that improves energy efficiency with minimal accuracy sacrifice. SpikingMamba integrates two key components: (a) SI-LIF, a signed-integer spiking neuron that preserves semantic polarity through signed multi-level spike representations. (b) A training-exclusive Smoothed Gradient Compensation (SGC) path mitigating quantization loss while preserving spike-driven efficiency. We employ a single-stage distillation strategy to transfer the zero-shot ability of pretrained Mamba and further enhance it via reinforcement learning (RL). Experiments show that SpikingMamba-1.3B achieves a 4.76$\times$ energy benefit, with only a 4.78\% zero-shot accuracy gap compared to the original Mamba. The model achieves a further 2.55\% accuracy improvement after RL, narrowing the performance gap from 4.78\% to 2.23\%.

PDF Details

AAAI Conference 2026 Conference Paper

Towards Privacy-Protected Generalized Gaze Estimation Using Diffusion Models and Domain Stability Adaptation Framework

Ziyi Wang
Shengcheng Ye
Faming Fang
Haichuan Song

Modern gaze estimation models can accurately predict human gaze from facial images. However, due to privacy concerns and intricate data collection procedures, gaze estimation datasets are typically smaller and less diverse compared to those for other vision tasks, which directly leads to poor generalization in gaze estimation models. Common solutions, such as domain adaptation models, require additional domain-specific data, yet such data is often difficult to obtain due to privacy restrictions. Meanwhile, domain generalization models suffer from limited performance due to insufficient training data. To address these fundamental challenges---privacy and data diversity---we explore privacy-preserving gaze data generation schemes and propose a novel data-driven generalization solution. Specifically, we develop two diffusion-based generative models, DDPM-Gaze and LDM-Gaze, for synthesizing gaze data. We demonstrate that synthetic data can significantly improve generalization performance when simply used with fine-tuning-based methods. Furthermore, we introduce the Domain Stability Adaptation (DSA) framework, a simple yet effective domain generalization approach that enhances model robustness by increasing the domain uncertainty of input samples while reducing prediction uncertainty. Extensive experiments validate the effectiveness of our synthetic data and demonstrate the superiority of our data-driven generalization solution.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Vision Transformers Are Circulant Attention Learners

Dongchen Han
Tianyu Li
Ziyi Wang
Gao Huang

The self-attention mechanism has been a key factor in the advancement of vision Transformers. However, its quadratic complexity imposes a heavy computational burden in high-resolution scenarios, restricting the practical application. Previous methods attempt to mitigate this issue by introducing handcrafted patterns such as locality or sparsity, which inevitably compromise model capacity. In this paper, we present a novel attention paradigm termed Circulant Attention by exploiting the inherent efficient pattern of self-attention. Specifically, we first identify that the self-attention matrix in vision Transformers often approximates the Block Circulant matrix with Circulant Blocks (BCCB), a kind of structured matrix whose multiplication with other matrices can be performed in O(NlogN) time. Leveraging this interesting pattern, we explicitly model the attention map as its nearest BCCB matrix and propose an efficient computation algorithm for fast calculation. The resulting approach closely mirrors vanilla self-attention, differing only in its use of BCCB matrices. Since our design is inspired by the inherent efficient paradigm, it not only delivers O(NlogN) computation complexity, but also largely maintains the capacity of standard self-attention. Extensive experiments on diverse visual tasks demonstrate the effectiveness of our approach, establishing circulant attention as a promising alternative to self-attention for vision Transformer architectures.

PDF Details DOI

EAAI Journal 2025 Journal Article

A Graph Convolutional Network and Gated Recurrent Unit-based surrogate for agent-based diffusion models

Yu Xiao
Yuanyuan Zhou
Ziyi Wang

This study addresses the challenge of high computational costs in agent-based diffusion models (ABMs), which are widely used for simulating complex diffusion processes but become prohibitively expensive in large-scale applications. To mitigate this issue, we introduce a Graph Convolutional Network (GCN) and Gated Recurrent Unit (GRU)-based Surrogate Network (G2SN) for ABMs. The GCN module captures the social network structure and seed set, while the GRU module models the diffusion time series. Computational complexity analysis demonstrates that G2SN significantly outperforms ABM simulations in efficiency. Experimental results confirm that G2SN accurately predicts ABM dynamics, reducing the mean absolute deviation (MAD) by 71. 7 % on training sets and 77. 7 % on test sets compared to traditional machine learning surrogate models. Case studies on new product diffusion further illustrate the effectiveness of the G2SN-based calibration approach, improving parameter search efficiency by 50. 8 % and 37. 2 % over alternative surrogate model-based methods. Additionally, these studies underscore the critical importance of social network and seed set in enhancing ABM prediction accuracy. This approach provides a more efficient and scalable tool for ABM calibration and new product diffusion forecasting, aiding managers in production, inventory, and marketing decisions.

Details DOI

JBHI Journal 2025 Journal Article

FCNCP: A Coupled Nonnegative CANDECOMP/PARAFAC Decomposition Based on Federated Learning

Yukai Cai
Hang Liu
Xiulin Wang
Hongjin Li
Ziyi Wang
Chuanshuai Yang
Fengyu Cong

Cognitive neuroscience is currently a field of research highly valued by many countries worldwide, and fostering corresponding international collaboration can accelerate the development of cognitive neuroscience in our country. However, challenges related to industry competition, privacy, and regulatory policies hinder international collaboration that relies on cross-server data sharing. Considering the current limitations of tensor decomposition methods in establishing constraints between cross-server data, this study leverages the advantages of federated learning to develop a federated non-negative coupled tensor decomposition framework (FCNCP), aimed at establishing coupling constraints across different servers while preserving privacy. In experiments validating the effectiveness of the algorithm, we conducted 50 decompositions on synthetic tensor data, achieving an average tensor fit coefficient of 0. 996, and the results demonstrated successful establishment of the coupling constraint. In real ERP data decomposition experiments, we applied the FCNCP algorithm to decompose ERP tensor data collected during proprioceptive stimulation applied to the left and right hands. The decomposition results revealed symmetrical activation areas in the left and right hemispheres induced by contralateral stimulation, with components in the beta and gamma frequency bands. These components are consistent with findings from related studies in cognitive neuroscience, confirming that this method can effectively handle high-dimensional EEG data across servers. This study not only provides new tools and approaches for processing and analyzing high-dimensional EEG data across servers but also promotes the advancement of coupled tensor decomposition techniques and their integration with emerging federated learning frameworks, offering significant theoretical and practical value.

Details DOI

NeurIPS Conference 2025 Conference Paper

Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials

Yifan Pu
Jixuan Ying
Qixiu Li
Tianzhu Ye
Dongchen Han
Xiaochen Wang
Ziyi Wang
shao xinyu

Vision Transformers (ViTs) have become a universal backbone for both image recognition and image generation. Yet their Multi–Head Self–Attention (MHSA) layer still performs a quadratic query–key interaction for \emph{every} token pair, spending the bulk of computation on visually weak or redundant correlations. We introduce \emph{Visual–Contrast Attention} (VCA), a drop-in replacement for MHSA that injects an explicit notion of discrimination while reducing the theoretical complexity from $\mathcal{O}(N^{2}C)$ to $\mathcal{O}(N n C)$ with $n\! \ll\! N$. VCA first distils each head’s dense query field into a handful of spatially pooled \emph{visual–contrast tokens}, then splits them into a learnable \emph{positive} and \emph{negative} stream whose differential interaction highlights what truly separates one region from another. The module adds fewer than $0. 3$\, M parameters to a DeiT-Tiny backbone, requires no extra FLOPs, and is wholly architecture-agnostic. Empirically, VCA lifts DeiT-Tiny top-1 accuracy on ImageNet-1K from $72. 2\%$ to \textbf{$75. 6\%$} (+$3. 4$) and improves three strong hierarchical ViTs by up to $3. 1$\%, while in class-conditional ImageNet generation it lowers FID-50K by $2. 1$ to $5. 2$ points across both diffusion (DiT) and flow (SiT) models. Extensive ablations confirm that (i) spatial pooling supplies low-variance global cues, (ii) dual positional embeddings are indispensable for contrastive reasoning, and (iii) combining the two in both stages yields the strongest synergy. VCA therefore offers a simple path towards faster and sharper Vision Transformers. The source code is available at \href{https: //github. com/LeapLabTHU/LinearDiff}{https: //github. com/LeapLabTHU/LinearDiff}.

PDF Details

NeurIPS Conference 2025 Conference Paper

LLM Safety Alignment is Divergence Estimation in Disguise

Rajdeep Haldar
Ziyi Wang
Guang Lin
Yue Xing
Qifan Song

We present a theoretical framework showing that popular LLM alignment methods—including RLHF and its variants—can be understood as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less-preferred) distributions. This perspective explains the emergence of separation in the latent space between safe and harmful prompts after alignment. As an application of our general divergence framework, we propose KLDO, a novel KL divergence-based alignment method, and empirically validate its effectiveness. We further show that using compliance–refusal datasets, rather than standard preference-based datasets, leads to stronger separation and improved safety alignment. Finally, to quantify the separation effect, we propose a distance-based metric in the prompt representation space, which also acts as a statistically significant indicator for model safety.

PDF Details

NeurIPS Conference 2024 Conference Paper

Demystify Mamba in Vision: A Linear Attention Perspective

Dongchen Han
Ziyi Wang
Zhuofan Xia
Yizeng Han
Yifan Pu
Chunjiang Ge
Jun Song
Shiji Song

Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba’s success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba’s success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-Inspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https: //github. com/LeapLabTHU/MLLA.

PDF Details DOI

TMLR Journal 2024 Journal Article

Enhancing Low-Precision Sampling via Stochastic Gradient Hamiltonian Monte Carlo

Ziyi Wang
Yujie Chen
Qifan Song
Ruqi Zhang

Low-precision training has emerged as a promising low-cost technique to enhance the training efficiency of deep neural networks without sacrificing much accuracy. Its Bayesian counterpart can further provide uncertainty quantification and improved generalization accuracy. This paper investigates low-precision sampling via Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) with low-precision and full-precision gradient accumulators for both strongly log-concave and non-log-concave distributions. Theoretically, our results show that to achieve $\epsilon$-error in the 2-Wasserstein distance for non-log-concave distributions, low-precision SGHMC achieves quadratic improvement ($\tilde{\mathcal{O}}\left({\epsilon^{-2}{\mu^*}^{-2}\log^2\left({\epsilon^{-1}}\right)}\right)$) compared to the state-of-the-art low-precision sampler, Stochastic Gradient Langevin Dynamics (SGLD) ($\tilde{\mathcal{O}}\left({{\epsilon}^{-4}{\lambda^{*}}^{-1}\log^5\left({\epsilon^{-1}}\right)}\right)$). Moreover, we prove that low-precision SGHMC is more robust to the quantization error compared to low-precision SGLD due to the robustness of the momentum-based update w.r.t. gradient noise. Empirically, we conduct experiments on synthetic data, and MNIST, CIFAR-10 \& CIFAR-100 datasets, which validate our theoretical findings. Our study highlights the potential of low-precision SGHMC as an efficient and accurate sampling method for large-scale and resource-limited machine learning.

PDF Details

EAAI Journal 2024 Journal Article

Multi-task label noise learning for classification

Zongmin Liu
Ziyi Wang
Ting Wang
Yitian Xu

Multi-task classification improves generalization performance via exploiting the correlations between tasks. However, most multi-task learning methods fail to recognize and filter noisy labels for the classification problems with label noises. To address this issue, this paper proposes a novel multi-task label noise learning method based on loss correction, called MTLNL. MTLNL introduces the class-wise denoising (CWD) method for loss decomposition and centroid estimation of the loss function in multi-task learning, and eliminates the impact of label noise by using label flipping rate. It also extends to the multi-task positive-unlabeled (PU) learning domain, which offers better flexibility and generalization performance. Moreover, Nesterov’s method is applied to accelerate the solution of the model. MTLNL is compared with other algorithms on five benchmark datasets, five image datasets, and a multi-task PU dataset to demonstrate its effectiveness.

Details DOI

NeurIPS Conference 2024 Conference Paper

SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection

Liangxin Liu
Xuebo Liu
Derek F. Wong
Dongfang Li
Ziyi Wang
Baotian Hu
Min Zhang

Instruction tuning (IT) is crucial to tailoring large language models (LLMs) towards human-centric interactions. Recent advancements have shown that the careful selection of a small, high-quality subset of IT data can significantly enhance the performance of LLMs. Despite this, common approaches often rely on additional models or data, which increases costs and limits widespread adoption. In this work, we propose a novel approach, termed $\textit{SelectIT}$, that capitalizes on the foundational capabilities of the LLM itself. Specifically, we exploit the intrinsic uncertainty present in LLMs to more effectively select high-quality IT data, without the need for extra resources. Furthermore, we introduce a curated IT dataset, the $\textit{Selective Alpaca}$, created by applying SelectIT to the Alpaca-GPT4 dataset. Empirical results demonstrate that IT using Selective Alpaca leads to substantial model ability enhancement. The robustness of SelectIT has also been corroborated in various foundation models and domain-specific tasks. Our findings suggest that longer and more computationally intensive IT data may serve as superior sources of IT, offering valuable insights for future research in this area. Data, code, and scripts are freely available at https: //github. com/Blue-Raincoat/SelectIT.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation

Ziyi Wang
Yanbo Wang
Xumin Yu
Jie Zhou
Jiwen Lu

Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at https: //github. com/wangzy22/XMask3D.

PDF Details DOI

NeurIPS Conference 2022 Conference Paper

P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Ziyi Wang
Xumin Yu
Yongming Rao
Jie Zhou
Jiwen Lu

Nowadays, pre-training big models on large-scale datasets has become a crucial topic in deep learning. The pre-trained models with high representation ability and transferability achieve a great success and dominate many downstream tasks in natural language processing and 2D vision. However, it is non-trivial to promote such a pretraining-tuning paradigm to the 3D vision, given the limited training data that are relatively inconvenient to collect. In this paper, we provide a new perspective of leveraging pre-trained 2D knowledge in 3D domain to tackle this problem, tuning pre-trained image models with the novel Point-to-Pixel prompting for point cloud analysis at a minor parameter cost. Following the principle of prompting engineering, we transform point clouds into colorful images with geometry-preserved projection and geometry-aware coloring to adapt to pre-trained image models, whose weights are kept frozen during the end-to-end optimization of point cloud analysis tasks. We conduct extensive experiments to demonstrate that cooperating with our proposed Point-to-Pixel Prompting, better pre-trained image model will lead to consistently better performance in 3D vision. Enjoying prosperous development from image pre-training field, our method attains 89. 3% accuracy on the hardest setting of ScanObjectNN, surpassing conventional point cloud models with much fewer trainable parameters. Our framework also exhibits very competitive performance on ModelNet classification and ShapeNet Part Segmentation. Code is available at https: //github. com/wangzy22/P2P.

PDF Details