Author name cluster

Jing Gu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

9 papers

2 author rows

EAAI Journal 2026 Journal Article

Inferring drug-related microbes through multi-perspective node feature distribution encoding and multi-scale hypergraph learning

Fengjiao Sun
Sentao Chen
Hui Cui
Jing Gu
Ping Xuan
Tiangang Zhang

Objective: Since microbes can affect the therapeutic effects of drugs by altering their structures, identifying drug-related microbes is crucial for understanding the mechanisms of drug actions. Recent research has focused on designing graph learning-based models to predict drug–microbe associations. However, these prediction methods failed to fully utilize the feature distributions of nodes within heterogeneous graphs comprising drugs and microbes. Previous methods failed to fully capture and leverage the complex correlations implied among multiple drugs and microbes. Additionally, the consistency of node representations across multiple graphs constructed from multiple views was also overlooked. To overcome these issues, we propose a Multi-perspective and Multi-scale drug–microbe Association prediction method, MMDMA, for predicting candidate microbes associated with various drugs. Methods: First, to propagate node information across multiple perspectives and learn feature distributions from the heterogeneous graphs, we develop a multi-perspective variational graph autoencoder. This component transfers node attributes between the functional and attribute perspectives, enabling feature learning for nodes across the heterogeneous graphs. Second, we design a hypergraph with dynamic topology, where each hyperedge encapsulates the biological characteristic of multiple microbes involved in the functional processes of multiple drugs. We propose a cross-attention mechanism that propagates node attributes to hyperedges, and this mechanism captures the inherent correlations between nodes and hyperedges, improving hyperedge attribute learning and dynamically evolving the hypergraph topology. Third, we develop a multi-scale hypergraph convolution framework to adaptively fuse the contextual relationships within the multi-scale neighbor topologies. Finally, given the diversity of node features for multiple heterogeneous graphs and the hypergraph, we design multi-graph contrastive learning to enhance the consistency of node features learnt from multiple graphs, ensuring robust node representation. Findings: The extensive comparison experiments demonstrate that MMDMA achieves area under the curve of 0. 967 and area under the precision–recall curve of 0. 883 and it outperforms nine state-of-the-art methods for predicting the candidate microbes for the interested drugs. Ablation studies validate the effectiveness of MMDMA’s major innovations, including local neighbor enhanced feature learning by multi-perspective variational graph autoencoder, multi-node correlation encoding by multi-scale hypergraph convolution, and multi-graph contrastive learning. Furthermore, case studies on three drugs confirm MMDMA’s capability to identify potential candidate microbes for diverse drugs. Novelty: MMDMA is capable of learning multi-perspective node feature distributions, encoding correlations among multiple nodes, and conducting contrastive learning of the node representations across multiple heterogeneous graphs and hypergraph. MMDMA can serve as an effective tool for screening reliable drug-related candidate microbes, which helps reduce the cost of experimentally discovering true drug–microbe associations in subsequent wet-lab studies. The source code and datasets are freely available at https: //github. com/pingxuan-hlju/MMDMA.

Details DOI

EAAI Journal 2025 Journal Article

A consistency regularization-based approach integrating anatomical structural relationships and organ category representations for multi-organ segmentation in pigs

Xiang Pan
Hang Fan
Jianlan Wang
Yan Fu
Wei Chu
Weipeng Tai
Jing Gu
Jianming Ni

In modern biomedical research and livestock management, accurate multi-organ segmentation in pigs is essential for breeding programs. However, current methods face challenges due to low imaging contrast, size disparities, and organ shape variability. Additionally, the manual annotation of computed tomography (CT) scans is labor-intensive and costly, limiting available labeled samples. To address these issues, we propose a consistency regularization-based network guided by anatomical structural relationships and global organ category representations, specifically designed for multi-organ segmentation using a limited number of annotated CT scan samples from pigs. Specifically, we designed the SpatialLink Gated Recurrent Unit (GRU) module to extract anatomical structural information and capture dynamic spatial relationships between organs, thereby minimizing segmentation biases caused by organ shape variations. Moreover, we developed the Organ Category Coding module and Guidance module, which integrate consistency regularization and attention mechanisms, enabling the network to accurately extract global organ category representations during the decoding phase, even with a small number of labeled samples, significantly improving segmentation consistency across organs of different sizes. Additionally, We are the first to apply the Visual State Space block to multi-organ segmentation in pigs, using it to extract contextual information. Experiments on 60 pigs demonstrate that our method achieves state-of-the-art results, with significant improvements in segmentation accuracy for the gallbladder and bladder, including a 9. 8% and 4. 2% Dice score increase, respectively, and a 12. 4% and 6. 2% boost in Jaccard scores compared to compared with a selection of published methods.

Details DOI

ICLR Conference 2025 Conference Paper

EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing

Kaizhi Zheng
Xiaotong Chen
Xuehai He
Jing Gu
Linjie Li
Zhengyuan Yang
Kevin Lin
Jianfeng Wang

Given the steep learning curve of professional 3D software and the time- consuming process of managing large 3D assets, language-guided 3D scene editing has significant potential in fields such as virtual reality, augmented reality, and gaming. However, recent approaches to language-guided 3D scene editing either require manual interventions or focus only on appearance modifications without supporting comprehensive scene layout changes. In response, we propose EditRoom, a unified framework capable of executing a variety of layout edits through natural language commands, without requiring manual intervention. Specifically, EditRoom leverages Large Language Models (LLMs) for command planning and generates target scenes using a diffusion-based method, enabling six types of edits: rotate, translate, scale, replace, add, and remove. To address the lack of data for language-guided 3D scene editing, we have developed an automatic pipeline to augment existing 3D scene synthesis datasets and introduced EditRoom-DB, a large-scale dataset with 83k editing pairs, for training and evaluation. Our experiments demonstrate that our approach consistently outperforms other baselines across all metrics, indicating higher accuracy and coherence in language-guided scene layout editing.

Details

NeSy Conference 2025 Conference Paper

JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents

Kaizhi Zheng
Kaiwen Zhou 0002
Jing Gu
Yue Fan
Jialu Wang
Zonglin Di
Xuehai He
Xin Eric Wang

Building a conversational embodied agent to execute real-life tasks has been a long-standing yet quite challenging research goal, as it requires effective human-agent communication, multi-modal understanding, long-range sequential decision making, etc. Traditional symbolic methods have scaling and generalization issues, while end-to-end deep learning models suffer from data scarcity and high task complexity, and are often hard to explain. To benefit from both worlds, we propose JARVIS, a neuro-symbolic commonsense reasoning framework for modular, generalizable, and interpretable conversational embodied agents. First, it acquires symbolic representations by prompting large language models (LLMs) for language understanding and sub-goal planning, and by constructing semantic maps from visual observations. Then the symbolic module reasons for sub-goal planning and action generation based on task- and action-level common sense. Extensive experiments on the TEACh dataset validate the efficacy and efficiency of our JARVIS framework, which achieves state-of-the-art (SOTA) results on all three dialog-based embodied tasks, including Execution from Dialog History (EDH), Trajectory from Dialog (TfD), and Two-Agent Task Completion (TATC) (e. g. , our method boosts the unseen Success Rate on EDH from 6. 1% to 15. 8%). Moreover, we systematically analyze the essential factors that affect the task performance and also demonstrate the superiority of our method in few-shot settings.

Details

ICLR Conference 2025 Conference Paper

Rethinking Multiple-Instance Learning From Feature Space to Probability Space

Zhaolong Du
Shasha Mao
Xuequan Lu
Mengnan Qi
Yimeng Zhang
Jing Gu
Licheng Jiao

Multiple-instance learning (MIL) was initially proposed to identify key instances within a set (bag) of instances when only one bag-level label is provided. Current deep MIL models mostly solve multi-instance problem in feature space. Nevertheless, with the increasing complexity of data, we found this paradigm faces significant risks in representation learning stage, which could lead to algorithm degradation in deep MIL models. We speculate that the degradation issue stems from the persistent drift of instances in feature space during learning. In this paper, we propose a novel Probability-Space MIL network (PSMIL) as a countermeasure. In PSMIL, a self-training alignment strategy is introduced in probability space to cope with the drift problem in feature space, and the alignment target objective is proven mathematically optimal. Furthermore, we reveal that the widely-used attention-based pooling mechanism in current deep MIL models is easily affected by the perturbation in feature space and further introduce an alternative called probability-space attention pooling. It effectively captures the key instance in each bag from feature space to probability space, and further eliminates the impact of selection drift in the pooling stage. To summarize, PSMIL seeks to solve a MIL problem in probability space rather than feature space. Experimental results illustrate that PSMIL could potentially achieve performance close to supervised learning level in complex tasks (gap within 5\%), with the incremental alignment in propability space bring more than 19\% accuracy improvements for current existing mainstream models in simulated CIFAR datasets. For existing publicly available MIL benchmarks/datasets, attention in probability space also achieves competitive performance to the state-of-the-art deep MIL models. Codes are available at \url{https://github.com/LMBDA-design/PSAMIL}.

Details

NeurIPS Conference 2025 Conference Paper

Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs

Fangrui Zhu
Hanhui Wang
Yiming Xie
Jing Gu
Tianye Ding
Jianwei Yang
Huaizu Jiang

Unlocking spatial reasoning in Multimodal Large Language Models (MLLMs) is crucial for enabling intelligent interaction with 3D environments. While prior efforts often rely on explicit 3D inputs or specialized model architectures, we ask: can MLLMs reason about 3D space using only structured 2D representations derived from perception? In this work, we introduce Struct2D, a perception-guided prompting framework that combines bird’s-eye-view (BEV) images with object marks and object-centric metadata, optionally incorporating egocentric keyframes when needed. Using Struct2D, we conduct an in-depth zero-shot analysis of closed-source MLLMs (e. g. , GPT-o3) and find that they exhibit surprisingly strong spatial reasoning abilities when provided with projected 2D inputs, effectively handling tasks such as relative direction estimation and route planning. Motivated by these findings, we construct a large-scale instructional tuning dataset, \textbf{Struct2D-Set}, using an automated pipeline that generates fine-grained QA pairs grounded in 3D indoor scenes. We then fine-tune an open-source MLLM (Qwen2. 5VL) using Struct2D-Set, relying on noisy 3D perception rather than ground-truth annotations. Despite this, the tuned model achieves strong performance across multiple spatial reasoning benchmarks, including 3D question answering, captioning, and object grounding, spanning eight diverse reasoning categories. Our approach demonstrates that structured 2D inputs can effectively bridge perception and language reasoning in MLLMs—without requiring explicit 3D representations as input. We will release both our code and dataset to support future research.

PDF Details

ICRA Conference 2024 Conference Paper

One-vs-All Semi-Automatic Labeling Tool for Semantic Segmentation in Autonomous Driving

Jing Gu
Guillermo Gallego 0002
Amine Ben Arab

Semantic image segmentation plays a pivotal role in creating High-Definition (HD) maps for autonomous driving, where every pixel in an image is assigned a label from a specific semantic class. However, obtaining dense pixel-level annotations for model training is a laborious and expensive process. Active learning holds promise as a method to reduce the human annotation effort needed for semantic segmentation. However, existing active learning methods often perform well in the majority classes but struggle with the minority classes, negatively impacting segmentation performance. To tackle this challenge, we propose a novel One-vs-All (OVA) active learning framework, known as OVAAL. This paper explains how OVAAL can shift more attention towards the minority classes and thoroughly analyzes its contributions to performance enhancement. Additionally, we introduce an OVA-based semi-supervised learning method as the final training phase, referred to as OVAAL+. Our results demonstrate that both OVAAL and OVAAL+ lead to significant improvements, with mean Intersection over Union (mIoU) gains of 4. 55% and 6. 38%, respectively, compared to the state-of-the-art active learning method Pixelpick on the Cityscapes semantic segmentation benchmark. These improvements are achieved while maintaining an economical annotation budget of 1. 44% of the training data. We foresee further research exploring the potential of OVA-based active selection to address challenges in cold start scenarios and resource-constrained training environments.

Details

NeurIPS Conference 2023 Conference Paper

PHOTOSWAP: Personalized Subject Swapping in Images

Jing Gu
Yilin Wang
Nanxuan Zhao
Tsu-Jui Fu
Wei Xiong
Qing Liu
Zhifei Zhang
He Zhang

In an era where images and visual content dominate our digital landscape, the ability to manipulate and personalize these images has become a necessity. Envision seamlessly substituting a tabby cat lounging on a sunlit window sill in a photograph with your own playful puppy, all while preserving the original charm and composition of the image. We present \emph{Photoswap}, a novel approach that enables this immersive image editing experience through personalized subject swapping in existing images. \emph{Photoswap} first learns the visual concept of the subject from reference images and then swaps it into the target image using pre-trained diffusion models in a training-free manner. We establish that a well-conceptualized visual subject can be seamlessly transferred to any image with appropriate self-attention and cross-attention manipulation, maintaining the pose of the swapped subject and the overall coherence of the image. Comprehensive experiments underscore the efficacy and controllability of \emph{Photoswap} in personalized subject swapping. Furthermore, \emph{Photoswap} significantly outperforms baseline methods in human ratings across subject swapping, background preservation, and overall quality, revealing its vast application potential, from entertainment to professional editing.

PDF Details

AAAI Conference 2021 Conference Paper

Perception Score: A Learned Metric for Open-ended Text Generation Evaluation

Jing Gu
Qingyang Wu
Zhou Yu

Automatic evaluation for open-ended natural language generation tasks remains a challenge. We propose a learned evaluation metric: Perception Score. It utilizes a pre-trained model and considers context information for conditional generation. Perception Score assigns a holistic score along with uncertainty measurement. We conduct experiments on three openended conditional generation tasks and two open-ended unconditional generation tasks. Perception Score achieves stateof-the-art results on all the tasks consistently in terms of correlation with human evaluation scores.

PDF Details