Bo Zhang

AAAI Conference 2026 Conference Paper

CareCom: Generative Image Composition with Calibrated Reference Features

• Jiaxuan Chen
• Bo Zhang
• Qingdong He
• Jinlong Peng
• Li Niu

Image composition aims to seamlessly insert foreground object into background. Despite the huge progress in generative image composition, the existing methods are still struggling with simultaneous detail preservation and foreground pose/view adjustment. To address this issue, we extend the existing generative composition model to multi-reference version, which allows using arbitrary number of foreground reference images. Furthermore, we propose to calibrate the global and local features of foreground reference images to make them compatible with the background information. The calibrated reference features can supplement the original reference features with useful global and local information of proper pose/view. Extensive experiments on MVImgNet and MureCom demonstrate that the generative model can greatly benefit from the calibrated reference features.

AAAI Conference 2026 Conference Paper

Counterfactual Question Generation Uncovering Learner Contradictions

• Bo Zhang
• Hao Yu
• Wenjie Dong
• Yvhang Yang
• Dezhuang Miao
• Fengyi Song
• Yanhui Gu
• Xiaoming Zhang

Conventional feedback, even when accompanied by brief explanations, rarely uncovers the hidden contradictions that trigger a learner's mistake. We bridge this gap with counterfactual question generation (CFQG): given a learner's answer, generate a follow-up question that deliberately contradicts it, compelling the learner to confront the underlying conflict. CFQG thus transforms assessment from passive scoring into an interactive and contradiction-centered dialogue that supports knowledge repair. To automate CFQG, we propose GapProbe, which probes the knowledge gap between a learner’s belief and curated facts through a knowledge graph (KG), then designs counterfactual questions (CFQs) that negate the belief. Identifying contradiction-aware triples, and more importantly, selecting those most likely to confuse the learner, are highly challenging in large-scale KGs. GapProbe tackles these challenges with an iterative ProConB cycle coupled with a schema-aware KGMap. By caching one- and multi-hop schema patterns of the KG, KGMap provides ``roadmap'' to guide LLMs jump to deep and contradiction-aware triples, beyond traditional step-wise graph traversal. We present the CFQG benchmark and corresponding metrics for evaluating how generated CFQs trigger, focus, and deepen learner reflection through explicit contradictions. Experiments on multiple datasets and LLMs show that GapProbe boosts LLM reasoning over KGs and generates follow-up questions that consistently promote deeper and more focused learner reflection.

AAAI Conference 2026 Conference Paper

Global-Local Confidence Fusion for Hallucination Detection in Mathematical Reasoning Task

• Bo Zhang
• Cong Gao
• Linkang Yang
• Bingxu Han
• Minghao Hu
• Zhunchen Luo
• Guotong Geng
• Xiaoying Bai

Large Reasoning Models (LRMs) achieve promising results on complex reasoning tasks but remain susceptible to hallucinations. Existing hallucination detection methods based on Large Language Models (LLMs) often focus solely on final answers, overlooking inconsistencies between the answer and reasoning process. This limitation reduces their ability to detect hallucinations during inference. Moreover, training-free approaches lack mechanisms for confidence estimation, resulting in an unquantified detection output. In contrast, training-based methods can provide fine-grained assessments but often neglect the self-correction capability of LRMs, where earlier errors may be corrected in subsequent steps, leading to inaccurate hallucination detection. To address these challenges, we propose ConfFuse, a unified framework that fuses global and local confidence scores for hallucination detection. A Global Hallucination Detection Model (GHDM) is trained using Direct Preference Optimization (DPO) to assess hallucinations at the level of entire reasoning chains, yielding global confidence estimates. Simultaneously, a Process Reward Model (PRM) estimates step-wise confidence scores to capture local logical flaws. A weighted fusion strategy combines the global confidence score with the minimum local score to jointly reflect overall reasoning consistency and local soundness. Experimental evaluations demonstrate that ConfFuse surpasses Qwen3-1.7B and Qwen3-8B by up to 11.86% and 5.46% in F1 score on in-distribution datasets, and achieves average improvements of 4.65% and 2.80% on out-of-distribution datasets. These results verify the effectiveness and generalizability of the proposed framework.

EAAI Journal 2026 Journal Article

Large language models for explainable fault diagnosis of machines

• Hamzah A.A.M. Qaid
• Bo Zhang
• Shuai Su
• Dan Li
• See-Kiong Ng
• Wei Li

Large Language Models (LLMs) have demonstrated remarkable capabilities in capturing complex conceptual representations from textual data for a wide range of real-world applications. However, in Intelligent Fault Diagnosis (IFD), leveraging sensor data such as vibration signals is essential but remains a challenge due to the modality gap between time series and LLMs’ inputs. Existing efforts to bridge this gap often treat LLMs merely as classifiers, overlooking their potential for understanding and reasoning over vibration-based data. In this paper, we propose a novel LLM-based fault diagnosis framework (FD-LLM) that aligns vibration signals with LLMs by encoding the signals into textual representations. FD-LLM introduces a classification-oriented approach, which formulates fault diagnosis as a multi-class classification task for benchmarking LLMs’ performance, and a context-aware spectrum language modeling approach that enables explainable, reasoning-driven fault analysis. We evaluate four open-source LLMs using FD-LLM across multiple datasets and noise conditions, assessing their validity, adaptability, and robustness. The results demonstrate that models such as LLaMA models achieve robust diagnostic performance, strong zero-shot adaptability across operating conditions, and effective generalization in cross-dataset scenarios with few-shot learning. The results further indicate that explainable fault diagnosis can be achieved in LLMs.

AAAI Conference 2026 Conference Paper

Learning to LEAP: Efficient Dense Point Tracking by Focusing Where It Matters

• Chenzhi Zhao
• Wufan Wang
• Bo Zhang
• Wendong Wang

Tracking Any Point (TAP) is a foundational task in computer vision with broad applicability. The state-of-the-art self-supervised TAP method leverages a global matching transformer and contrastive random walks to learn point correspondences. However, its dense all-pairs attention and correlation volume computation tend to introduce irrelevant features and produce less informative training signals, degrading both learning efficiency and tracking accuracy. To address these limitations, we introduce LEAP-Track, a self-supervised TAP approach that computes the attention matrices and correlation volume over adaptively selected sparse pairs. It consists of two core designs: (1) Curriculum-based Sparse Attention (CSA), which dynamically focuses on the most relevant keys, promoting the learning of discriminative features; and (2) Progressive k-NN Transition (PkT), which reformulates the contrastive random walk to operate on an increasingly sparse k-NN affinity graph to reinforce the learning of the most informative correspondences. By integrating the above two designs into a two-stage training paradigm, LEAP-Track is shown both theoretically and empirically to effectively boost learning efficiency, achieving superior tracking accuracy over existing self-supervised TAP methods.

AAMAS Conference 2026 Conference Paper

MeCo: Enhancing LLM-Empowered Multi-Robot Collaboration via Similar Task Memoization

• Baiqing Wang
• Helei Cui
• Bo Zhang
• Xiaolong Zheng
• Bin Guo
• Zhiwen Yu

Multi-robot systems have been widely deployed in real-world applications, providing significant improvements in efficiency and reductions in labor costs. However, most existing multi-robot collaboration methods rely on extensive task-specific training, which limitstheiradaptabilitytonewordiversescenarios. Recentresearch leverages the language understanding and reasoning capabilities of large language models (LLMs) to enable more flexible collaboration without specialized training. Yet, current LLM-empowered approaches remain inefficient: when confronted with identical or similar tasks, they must replan from scratch because they omit tasklevel similarities. To address this limitation, we propose MeCo, a similarity-aware multi-robot collaboration framework that applies the principle of “cache and reuse” (a. k. a. , memoization) to reduce redundant computation. Unlike simple task repetition, identifying and reusing solutions for similar but not identical tasks is far more challenging, particularly in multi-robot settings. To this end, MeCo introduces a new similarity testing method that retrieves previouslysolvedtaskswithhighrelevance, enablingeffectiveplanreuse without re-invoking LLMs. Furthermore, we present MeCoBench, the first benchmark designed to evaluate performance on similartask collaboration scenarios. Experimental results show that MeCo substantially reduces planning costs and improves success rates compared with state-of-the-art approaches.

AAAI Conference 2026 Conference Paper

Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports from Scratch with Agentic Framework

• Zhaorui Yang
• Bo Pan
• Han Wang
• Yiyao Wang
• Xingyu Liu
• Luoxuan Weng
• Yingchaojie Feng
• Haozhe Feng

Visualizations play a crucial part in effective communication of concepts and information. Recent advances in reasoning and retrieval augmented generation have enabled Large Language Models (LLMs) to perform deep research and generate comprehensive reports. Despite its progress, existing deep research frameworks primarily focus on generating text-only content, leaving the automated generation of interleaved texts and visualizations underexplored. This novel task poses key challenges in designing informative visualizations and effectively integrating them with text reports. To address these challenges, we propose Formal Description of Visualization (FDV), a structured textual representation of charts that enables LLMs to learn from and generate diverse, high-quality visualizations. Building on this representation, we introduce Multimodal DeepResearcher, an agentic framework that decomposes the task into four stages: (1) researching, (2) exemplar report textualization, (3) planning and (4) multimodal report generation. For the evaluation of the generated reports, we develop MultimodalReportBench which contains 100 diverse topics as inputs, and a set of dedicated metrics for report and chart evaluation. Extensive experiments across models and evaluation methods demonstrate the effectiveness of Multimodal DeepResearcher. Notably, utilizing the same Claude 3.7 Sonnet model, Multimodal DeepResearcher achieves an 82% overall win rate over the baseline method.

AAAI Conference 2026 Conference Paper

Semore: VLM-guided Enhanced Semantic Motion Representations for Visual Reinforcement Learning

• Wentao Wang
• Chunyang Liu
• Kehua Sheng
• Bo Zhang
• Yan Wang

The growing exploration of Large Language Models (LLM) and Vision-Language Models (VLM) has opened avenues for enhancing the effectiveness of reinforcement learning (RL). However, existing LLM-based RL methods often focus on the guidance of control policy and encounter the challenge of limited representations of the backbone networks. To tackle this problem, we introduce Enhanced Semantic Motion Representations (Semore), a new VLM-based framework for visual RL, which can simultaneously extract semantic and motion representations through a dual-path backbone from the RGB flows. Semore utilizes VLM with common-sense knowledge to retrieve key information from observations, while using the pre-trained clip to achieve the text-image alignment, thereby embedding the ground-truth representations into the backbone. To efficiently fuse semantic and motion representations for decision-making, our method adopts a separately supervised approach to simultaneously guide the extraction of semantics and motion, while allowing them to interact spontaneously. Extensive experiments demonstrate that, under the guidance of VLM at the feature level, our method exhibits efficient and adaptive ability compared to state-of-the-art methods. All codes are released.

AAAI Conference 2026 Conference Paper

ViG-RAG: Video-aware Graph Retrieval-Augmented Generation via Temporal and Semantic Hybrid Reasoning

• Zongsheng Cao
• Anran Liu
• Yangfan He
• Jing Li
• Bo Zhang
• Zigan Wang

Retrieval-augmented generation (RAG) has greatly improved Large Language Models (LLMs) by adding external knowledge. However, current RAG-based methods face difficulties with long-context video understanding due to two main challenges. First, Current RAG-based methods for long-context video understanding struggle to effectively integrate multimodal and long-range temporal information, resulting in fragmented and context-insensitive knowledge representations. Furthermore, their retrieval mechanisms often rely on static textual matching, failing to dynamically align user queries with the most relevant video segments and leading to suboptimal downstream performance. To overcome these issues, we introduce ViG-RAG, a new framework to enhance long-context video understanding through structured textual knowledge grounding and multi-modal retrieval. Specifically, we segment video transcripts into structured units, extract key entities, form temporal connections, and assign confidence for evidence, enabling coherent long-range reasoning. In this way, it utilizes a knowledge-aware grounding mechanism and a context-aware retrieval process that dynamically builds a probabilistic temporal knowledge graph to organize multi-video content. To improve retrieval accuracy, we propose a hybrid retrieval strategy for semantic and temporal features, with an adaptive distribution modeling the relevance. In this way, it achieves the optimal retrieval distribution for each query, enhancing generation efficiency by reducing unnecessary computations. On top of this, ViG-RAG uses a vision-language model to integrate semantic anchors, expanded contextual fields, and selected video frames, generating an accurate response. We evaluate ViG-RAG on several benchmarks, demonstrating that it significantly surpasses current RAG-based methods.

EAAI Journal 2025 Journal Article

A prior segmentation knowledge enhanced deep learning system for the classification of tumors in ultrasound image

• Tao Jiang
• Jun Guo
• Wenyu Xing
• Ming Yu
• Yifang Li
• Bo Zhang
• Yi Dong
• Dean Ta

Breast and thyroid cancers are prevalent among women worldwide. Ultrasound (US) examination is widely used for the early detection of breast and thyroid cancers. However, due to the blurred tumor boundaries and irregular shapes, the computer-aided diagnosis (CAD) of tumors based on US is challenging. Numerous studies have introduced deep learning-based multi-task learning approaches to address this issue, but these methods may result in feature redundancy and misinformation. Tumor segmentation is a prerequisite step for US CAD, and a higher Dice coefficient is associated with more accurate classification outcomes. Therefore, this paper introduces a novel deep-learning system that fully utilizes segmentation knowledge to boost classification performance. The system starts with a hybrid convolutional neural network (CNN)-Transformer for tumor localization and coarse segmentation, then uses a lightweight CNN-based U-Net to refine segmentation results. Subsequently, segmentation knowledge is harnessed to augment the network input and enhance multimodal feature extraction, resulting in improved classification performance. Our proposed method yielded a Dice coefficient of 83. 62% and 77. 20% for breast and thyroid tumor segmentation and area under curve (AUC) values of 0. 9536 and 0. 9475 for their respective classifications. Compared to non-segmentation knowledge-based classification models, our method obtained an increase in AUC of 0. 1054 and 0. 0566 on the breast and thyroid datasets, respectively. It outperformed the performance of the State-Of-The-Art (SOTA) methods across various datasets. In summary, our proposed system shows promise for application in US tumor analysis and holds potential to be extended to additional diseases and modalities.

IROS Conference 2025 Conference Paper

Bench4Merge: A Comprehensive Benchmark for Merging in Realistic Dense Traffic with Micro-Interactive Vehicles

• Zhengming Wang
• Junli Wang
• Pengfei Li
• Zhaohan Li
• Chunyang Liu
• Bo Zhang
• Peng Li
• Yilun Chen

While the capabilities of autonomous driving have advanced rapidly, merging into dense traffic remains a significant challenge, many motion planning methods for this scenario have been proposed but it is hard to evaluate them. Most existing closed-loop simulators rely on rule-based controls for other vehicles, which results in a lack of diversity and randomness, thus failing to accurately assess the motion planning capabilities in highly interactive scenarios. Moreover, traditional evaluation metrics are insufficient for comprehensively evaluating the performance of merging in dense traffic. In response, we proposed a closed-loop evaluation benchmark for assessing motion planning capabilities in merging scenarios. Our approach involves other vehicles trained in large scale datasets with micro-behavioral characteristics that significantly enhance the complexity and diversity. Additionally, we have restructured the evaluation mechanism by leveraging Large Language Models (LLMs) to assess each autonomous vehicle merging onto the main lane. Extensive experiments and test-vehicle deployment have demonstrated the progressiveness of this benchmark. Through this benchmark, we have obtained an evaluation of existing methods and identified common issues. The simulation environment and evaluation process can be accessed at https://github.com/WZM5853/Bench4Merge.

NeurIPS Conference 2025 Conference Paper

Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression

• Xiaohui Wang
• Peng Ye
• Chenyu Huang
• Shenghe Zheng
• Bo Zhang
• Lei Bai
• Wanli Ouyang
• Tao Chen

With the rise of the fine-tuned–pretrained paradigm, storing numerous fine-tuned models for multi-tasking creates significant storage overhead. Delta compression alleviates this by storing only the pretrained model and the highly compressed delta weights (the differences between fine-tuned and pretrained model weights). However, existing methods fail to maintain both high compression and performance, and often rely on data. To address these challenges, we propose UltraDelta, the first data-free delta compression pipeline that achieves both ultra-high compression and strong performance. UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer, and global dimensions, using three key components: (1) Variance-Based Mixed Sparsity Allocation assigns sparsity based on variance, giving lower sparsity to high-variance layers to preserve inter-layer information. (2) Distribution-Aware Compression applies uniform quantization and then groups parameters by value, followed by group-wise pruning, to better preserve intra-layer distribution. (3) Trace-Norm-Guided Rescaling uses the trace norm of delta weights to estimate a global rescaling factor, improving model stability under higher compression. Extensive experiments across (a) large language models (fine-tuned on LLaMA-2 7B and 13B) with up to 50$\times$ compression, (b) general NLP models (RoBERTa-base, T5-base) with up to 224$\times$ compression, (c) vision models (ViT-B/32, ViT-L/14) with up to 132$\times$ compression, and (d) multi-modal models (BEiT-3) with 18$\times$ compression, demonstrate that UltraDelta consistently outperforms existing methods, especially under ultra-high compression. Code is available at https: //github. com/xiaohuiwang000/UltraDelta.

IROS Conference 2025 Conference Paper

DGVO: A Dynamically Constrained Gradient Velocity Obstacle Approach for Mobile Robots in Dynamic Environments

• Bowen Xiao
• Bo Zhang
• Danyu Zhang
• Peiyan Xie
• Xinyu Wang
• Ruocheng Li

In this paper, we propose a framework based on velocity obstacles to address dynamic obstacle avoidance problem for constrained mobile robots. The framework establishes a nonlinear mapping from the control domain to the velocity space based on the robot’s kinematic model and input constraints. This mapping defines the Velocity Feasible Region (VFR) as the set of reachable velocities at the next time step. Utilizing the VFR, we propose a gradient field, called the Dynamically Constrained Gradient Velocity Obstacle (DGVO), to represent the feasible motion region for mobile robots. DGVO preserves the original feasible region of the mobile robot. Based on DGVO, we formulate an unconstrained gradient descent optimization problem to compute collision-free velocities in real time. This framework enables real-time online computation of collision-free velocities for any constrained mobile robot, and it exhibits strong robustness to sensor noise. Extensive simulations and real-world experiments have validated the effectiveness of the proposed method. The introduction of the entire work can be found at the following link: https://youtu.be/HrTNTSOhKvE.

AAAI Conference 2025 Conference Paper

DiffCalib: Reformulating Monocular Camera Calibration as Diffusion-Based Dense Incident Map Generation

• Xiankang He
• Guangkai Xu
• Bo Zhang
• Hao Chen
• Ying Cui
• Dongyan Guo

Monocular camera calibration is a key precondition for numerous 3D vision applications. Despite considerable advancements, existing methods often hinge on specific assumptions and struggle to generalize across varied real-world scenarios, and the performance is limited by insufficient training data. Recently, diffusion models trained on expansive datasets have been confirmed to maintain the capability to generate diverse, high-quality images. This success suggests a strong potential of the models to effectively understand varied visual information. In this work, we leverage the comprehensive visual knowledge embedded in pre-trained diffusion models to enable more robust and accurate monocular camera intrinsic estimation. Specifically, we reformulate the problem of estimating the four degrees of freedom (4-DoF) of camera intrinsic parameters as a dense incident map generation task. The map details the angle of incidence for each pixel in the RGB image, and its format aligns well with the paradigm of diffusion models. The camera intrinsic then can be derived from the incident map with a simple non-learning RANSAC algorithm during inference. Moreover, to further enhance the performance, we jointly estimate a depth map to provide extra geometric information for the incident map estimation. Extensive experiments on multiple testing datasets demonstrates that our model achieves state-of-the-art performance, gaining up to a 40% reduction in prediction errors. Besides, the experiments also show that the precise camera intrinsic and depth maps estimated by our pipeline can greatly benefit practical applications such as 3D reconstruction from a single in-the-wild image.

EAAI Journal 2025 Journal Article

Enhanced air pollution spatiotemporal forecast model using frequency domain convolution and attention mechanism

• Haiwei Yang
• Ru Yang
• Ling Ding
• Shiqiang Du
• Maozhen Li
• Bo Zhang

Accurate forecasting of air pollution, which is crucial for public health and environmental management, often faces challenges in effectively capturing the complex and intertwined spatiotemporal dynamics of pollutants. Existing models frequently struggle to simultaneously account for broad periodic spatiotemporal dependencies as well as fine-grained local temporal patterns. This paper presents a novel deep learning architecture, the Fourier Convolutional Graph Transformer (FCGformer), specifically designed to overcome these limitations. FCGformer distinctively features a dual-module approach: a Global Module that constructs an integrated spatiotemporal graph and leverages Fourier transforms with frequency domain convolution to extract long-range dependencies and crucial periodicities; and a Local Module that employs inverse temporal embedding and self-attention to meticulously capture nuanced, short-term temporal variations. The key contribution of this work lies in the synergistic integration that enables FCGformer to effectively model complex pollutant behaviors, providing a more comprehensive understanding of both global contexts and local details. Extensive experiments demonstrate that FCGformer significantly outperforms state-of-the-art benchmark models in prediction accuracy, offering a promising advancement for improved air quality management.

AAAI Conference 2025 Conference Paper

LiON: Learning Point-Wise Abstaining Penalty for LiDAR Outlier DetectioN Using Diverse Synthetic Data

• Shaocong Xu
• Pengfei Li
• Qianpu Sun
• Xinyu Liu
• Yang Li
• Shihui Guo
• Zhen Wang
• Bo Jiang

LiDAR-based semantic scene understanding is an important module in the modern autonomous driving perception stack. However, identifying outlier points in a LiDAR point cloud is challenging as LiDAR point clouds lack semantically-rich information. While former SOTA methods adopt heuristic architectures, we revisit this problem from the perspective of Selective Classification, which introduces a selective function into the standard closed-set classification setup. Our solution is built upon the basic idea of abstaining from choosing any inlier categories but learns a point-wise abstaining penalty with a margin-based loss. Apart from learning paradigms, synthesizing outliers to approximate unlimited real outliers is also critical, so we propose a strong synthesis pipeline that generates outliers originated from various factors: object categories, sampling patterns and sizes. We demonstrate that learning different abstaining penalties, apart from point-wise penalty, for different types of (synthesized) outliers can further improve the performance. We benchmark our method on SemanticKITTI and nuScenes and achieve SOTA results.

NeurIPS Conference 2025 Conference Paper

TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception

• Runjian Chen
• Hyoungseob Park
• Bo Zhang
• Wenqi Shao
• Ping Luo
• Alex Wong

Labeling LiDAR point clouds is notoriously time-and-energy-consuming, which spurs recent unsupervised 3D representation learning methods to alleviate the labeling burden in LiDAR perception via pretrained weights. Existing work focus on either masked auto encoding or contrastive learning on LiDAR point clouds, which neglects the temporal LiDAR sequence that naturally accounts for object motion (and their semantics). Instead, we propose TREND, short for Temporal REndering with Neural fielD, to learn 3D representation via forecasting the future observation in an unsupervised manner. TREND integrates forecasting for 3D pre-training through a Recurrent Embedding scheme to generate 3D embeddings across time and a Temporal LiDAR Neural Field specifically designed for LiDAR modality to represent the 3D scene, with which we compute the loss using differentiable rendering. We evaluate TREND on 3D object detection and LiDAR semantic segmentation tasks on popular datasets, including Once, Waymo, NuScenes, and SemanticKITTI. TREND generally improves from-scratch models across datasets and tasks and brings gains of 1. 77\% mAP on Once and 2. 11\% mAP on NuScenes, which are up to 400\% more improvement compared to previous SOTA unsupervised 3D pre-training methods. Codes and models will be available.

EAAI Journal 2025 Journal Article

Uncertainty-aware focal loss for object segmentation

• Lei Chen
• Yang Wang
• Jibin Yang
• Yunfei Zheng
• Tong Han
• Bo Zhang
• Tieyong Cao

In the loss function of object segmentation models, misclassified pixels whose prediction are opposite to the ground truth and uncertain pixels whose predicted probability is close to 0. 5 are more important for model training. Focusing on misclassified pixels can improve the segmentation accuracy of the model, and focusing on uncertain pixels can help the model to form better decision surfaces. However, existing methods fail to take both types of pixels into account simultaneously. To enhance the learning on these two types of important pixels, the Uncertainty-aware Focal Loss (UFL) is proposed based on the analysis of Uncertainty-aware Loss (UAL). Then, by leveraging the S-shaped property of the sigmoid function, a loss function is constructed that can simultaneously increase the loss and loss derivatives of misclassified and uncertain pixels. In order to solve the gradient vanishing problem of the sigmoid function on well-classified pixels, a regularization constraint term is defined, whose value is the square of predicted probability. Finally, the pixel loss value is dynamically adjusted at different stages of training according to the changes in the contributions of misclassified and uncertain pixels to the model training, which improves the targeted learning for misclassified and uncertain pixels. Experimental results on two different types of network structures and six datasets demonstrate that the proposed method can better segment the uncertain and misclassified pixels. Especially, on the DUT-O dataset, UFL improves mean Intersection over Union (mIoU) by almost 2. 7 % compared to UAL.

IROS Conference 2025 Conference Paper

VDTF-ACT: ACT-based Multimodal Space Fine Manipulation Method with Visual Depth Tactile Fusion

• Siyi Lang
• Jihang Chen
• Bo Zhang
• Hanlin Dong
• Panfeng Huang
• Zhiqiang Ma 0001

Autonomous fine manipulation in space for orbital assembly continues to present a critical challenge in the field of aerospace engineering. Under low-gravity conditions, during satellite manipulator operations on free-floating objects, the absence of significant gravitational forces and friction constraints leads to unpredictable relative motions between the manipulator’s end-effectors and objects, degrading the manipulation performance. This study proposes a fine manipulation method for satellite robots with floating platforms, grounded in multimodal perception and an enhanced Action Chunking with Transformer (ACT) architecture that enables multimodal state interaction. By integrating visual, tactile, and depth sensory data, the satellite robot’s space fine manipulation capabilities are substantially improved. From the experimental results generated from simulated environments, the proposed method achieves a success rate exceeding 80% for peg-in-socket insertion tasks, outperforming conventional approaches with a success rate of approximately 45%. Project Website: https://github.com/LSY0528/VDTF-ACT.

AAAI Conference 2025 Conference Paper

What Is a Good Question? Assessing Question Quality via Meta-Fact Checking

• Bo Zhang
• Jianghua Zhu
• Chaozhuo Li
• Hao Yu
• Li Kong
• Zhan Wang
• Dezhuang Miao
• Xiaoming Zhang

Knowledge-based questions are typically employed to evaluate LLM's knowledge boundaries; meanwhile, numerous studies focus on question generation as a means to enhance the capabilities of both models and individuals. However, there is a lack of in-depth exploration about what constitutes a good question from the perspective of knowledge cognition. This paper proposes aligning the complete knowledge underlying questions with educational criteria effectively employed in physics courses, thereby developing novel knowledge-intensive metrics of question quality. To this end, we propose Meta-Fact Checking (MFC), which transforms questions into knowledge graph (KG) triples utilizing LLMs through few-shot prompting, thereby quantifying question quality based on the patterns observed within these triples. MFC introduces a novel interaction mechanism for KGs that communicates meta-facts, illustrating the types of knowledge that KGs can offer to the LLM for reasoning questions, rather than relying solely on the original triples. This strategy ensures that MFC remains unaffected by unexplored triples that LLM has not yet encountered within KGs compared to the retrieve-while-reasoning routine. Experiments across multiple datasets and LLMs demonstrate that MFC significantly improves the accuracy and efficiency of both question answering and assessing. This research marks a pioneering effort to automate the evaluation of question quality based on cognitive capabilities.

EAAI Journal 2024 Journal Article

A commercial vehicle weight prediction method based on driving simulation data

• Yu Liu
• Hao Zhang
• Xianglei Zhu
• Jingyuan Li
• Hang Xu
• Bo Zhang
• Hanzhengnan Yu
• Yu Wang

Neural network is beneficial for obtaining weight, which is a crucial parameter for vehicle in operation. However, the high demand of neural network for real data, and the lack of in-depth research on multi-vehicle weight prediction scenarios, have limited the development of weight prediction. In this paper, relying on the genetic algorithm-optimized BP neural network (GA-BP neural network), a commercial vehicle weight prediction method based on driving simulation data was proposed. The driving data of multiple commercial vehicle types were collected and used to study the weight prediction effect of the proposed method in single-vehicle and multi-vehicle scenarios. The fuzzy processing method of driving simulation data was proposed to process simulation data instead of real data to train the model, which effectively solves the problem of high demand for real data in neural network method. The study shows that proposed method can achieve weight prediction deviation of 2. 87% for single-vehicle model and 4. 63% for multi-vehicle model.

EAAI Journal 2024 Journal Article

A reliable traversability learning method based on human-demonstrated risk cost mapping for mobile robots over uneven terrain

• Bo Zhang
• Guobin Li
• Jiale Zhang
• Xiaoshan Bai

The paper proposed a traversability learning method based on the human demonstration for generating risk cost maps. These maps aid mobile robots in identifying safe areas for reliable autonomous navigation over uneven terrain. Firstly, a maximum causal entropy-based inverse reinforcement learning method is employed to generate a reward function by considering human-demonstrated trajectories, robot poses, and feature vectors extracted from elevation data. This reward function is intended to accurately capture the behavioral preferences identified in human-demonstrated trajectories, specifically focusing on low-risk areas of the environment. Secondly, the reward function is combined with terrain feature data to generate a cost map and least-cost trajectory. Utilizing a wheeled mobile robot traversing uneven terrain, this paper verifies the adaptability enhancement of the proposed method for autonomous navigation over outdoor uneven terrain. The experimental results show an increase of 4%–10% in the success rate, a decrease of 13. 6%–32. 1% in the cumulative slope and gradient, and a decrease of 20. 8%–27. 4% in the Hausdorff distance of the robot’s trajectories compared with traditional inverse reinforcement learning-based navigation methods.

EAAI Journal 2024 Journal Article

Balancing therapeutic effect and safety in ventilator parameter recommendation: An offline reinforcement learning approach

• Bo Zhang
• Xihe Qiu
• Xiaoyu Tan

Reinforcement learning (RL) is increasingly applied in recommending ventilator parameters, yet existing methods prioritize therapeutic effect over patient safety. This leads to excessive exploration by the RL agent, posing risks. To address this, we propose a novel offline RL approach that leverages existing clinical data for exploration, employing fitted Q evaluation (FQE) for policy evaluation to minimize patient risk compared to online evaluation. Our method introduces a variational auto-encoder-gumble softmax (VAE-GS) model, discerning the hidden relationship between patient physiological status and ventilator parameters to constrain the exploratory space of the agent. Additionally, a noise network aids the agent in fully exploring the reachable space to find optimal ventilator parameters. Our approach significantly enhances safety, as evidenced by experiments on the Medical Information Mart for Intensive Care III (MIMIC-III) dataset. It outperforms existing algorithms including the deterministic policy gradient algorithm (DDPG), soft actor-critic (SAC), batch-constrained deep Q-learning (BCQ), conservative Q-learning (CQL), and closed-form policy improvement operators (CFPI), showing improvements of 76. 9%, 82. 8%, 23. 5%, 49. 1% and 23. 5%, respectively, while maintaining therapeutic effect.

NeurIPS Conference 2024 Conference Paper

Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving

• Jianbiao Mei
• Yukai Ma
• Xuemeng Yang
• Licheng Wen
• Xinyu Cai
• Xin Li
• Daocheng Fu
• Bo Zhang

Autonomous driving has advanced significantly due to sensors, machine learning, and artificial intelligence improvements. However, prevailing methods struggle with intricate scenarios and causal relationships, hindering adaptability and interpretability in varied environments. To address the above problems, we introduce LeapAD, a novel paradigm for autonomous driving inspired by the human cognitive process. Specifically, LeapAD emulates human attention by selecting critical objects relevant to driving decisions, simplifying environmental interpretation, and mitigating decision-making complexities. Additionally, LeapAD incorporates an innovative dual-process decision-making module, which consists of an Analytic Process (System-II) for thorough analysis and reasoning, along with a Heuristic Process (System-I) for swift and empirical processing. The Analytic Process leverages its logical reasoning to accumulate linguistic driving experience, which is then transferred to the Heuristic Process by supervised fine-tuning. Through reflection mechanisms and a growing memory bank, LeapAD continuously improves itself from past mistakes in a closed-loop environment. Closed-loop testing in CARLA shows that LeapAD outperforms all methods relying solely on camera input, requiring 1-2 orders of magnitude less labeled data. Experiments also demonstrate that as the memory bank expands, the Heuristic Process with only 1. 8B parameters can inherit the knowledge from a GPT-4 powered Analytic Process and achieve continuous performance improvement. Project page: https: //pjlab-adg. github. io/LeapAD

NeurIPS Conference 2024 Conference Paper

DomainGallery: Few-shot Domain-driven Image Generation by Attribute-centric Finetuning

• Yuxuan Duan
• Yan Hong
• Bo Zhang
• Jun Lan
• Huijia Zhu
• Weiqiang Wang
• Jianfu Zhang
• Li Niu

The recent progress in text-to-image models pretrained on large-scale datasets has enabled us to generate various images as long as we provide a text prompt describing what we want. Nevertheless, the availability of these models is still limited when we expect to generate images that fall into a specific domain either hard to describe or just unseen to the models. In this work, we propose DomainGallery, a few-shot domain-driven image generation method which aims at finetuning pretrained Stable Diffusion on few-shot target datasets in an attribute-centric manner. Specifically, DomainGallery features prior attribute erasure, attribute disentanglement, regularization and enhancement. These techniques are tailored to few-shot domain-driven generation in order to solve key issues that previous works have failed to settle. Extensive experiments are given to validate the superior performance of DomainGallery on a variety of domain-driven generation scenarios.

ICLR Conference 2024 Conference Paper

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior

• Jingxiang Sun
• Bo Zhang
• Ruizhi Shao
• Lizhen Wang 0002
• Wen Liu
• Zhenda Xie
• Yebin Liu

We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose bootstrapped score distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

NeurIPS Conference 2024 Conference Paper

Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks

• Xin-Chun Li
• Jin-Lin Tang
• Bo Zhang
• Lan Li
• De-Chuan Zhan

Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the {\it degree of sign consistency} between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.

AAAI Conference 2024 Conference Paper

LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection

• Hongcheng Guo
• Jian Yang
• Jiaheng Liu
• Jiaqi Bai
• Boyang Wang
• Zhoujun Li
• Tieqiao Zheng
• Bo Zhang

Log anomaly detection is a key component in the field of artificial intelligence for IT operations (AIOps). Considering log data of variant domains, retraining the whole network for unknown domains is inefficient in real industrial scenarios. However, previous deep models merely focused on extracting the semantics of log sequences in the same domain, leading to poor generalization on multi-domain logs. To alleviate this issue, we propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains, where we establish a two-stage process including the pre-training and adapter-based tuning stage. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters. Besides, the Log-Attention module is proposed to supplement the information ignored by the log-paring. The proposed method is evaluated on three public datasets and one real-world dataset. Experimental results on multiple benchmarks demonstrate the effectiveness of our LogFormer with fewer trainable parameters and lower training costs.

AAAI Conference 2024 Conference Paper

Make RepVGG Greater Again: A Quantization-Aware Approach

• Xiangxiang Chu
• Liang Li
• Bo Zhang

The tradeoff between performance and inference speed is critical for practical applications. Architecture reparameterization obtains better tradeoffs and it is becoming an increasingly popular ingredient in modern convolutional neural networks. Nonetheless, its quantization performance is usually too poor to deploy (e.g. more than 20% top-1 accuracy drop on ImageNet) when INT8 inference is desired. In this paper, we dive into the underlying mechanism of this failure, where the original design inevitably enlarges quantization error. We propose a simple, robust, and effective remedy to have a quantization-friendly structure that also enjoys reparameterization benefits. Our method greatly bridges the gap between INT8 and FP32 accuracy for RepVGG. Without bells and whistles, the top-1 accuracy drop on ImageNet is reduced within 2% by standard post-training quantization. Extensive experiments on detection and semantic segmentation tasks verify its generalization.

AAAI Conference 2024 Conference Paper

Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models

• Liang Li
• Qingyuan Li
• Bo Zhang
• Xiangxiang Chu

As the size of large language models (LLMs) continues to grow, model compression without sacrificing accuracy has become a crucial challenge for deployment. While some quantization methods, such as GPTQ, have made progress in achieving acceptable 4-bit weight-only quantization, attempts at lower-bit quantization often result in severe performance degradation. In this paper, we introduce a technique called norm tweaking, which can be used as a plugin in current PTQ methods to achieve high precision while being cost-efficient. Our approach is inspired by the observation that rectifying the quantized activation distribution to match its float counterpart can readily restore accuracy for LLMs. To achieve this, we carefully design a tweaking strategy that includes calibration data generation and channel-wise distance constraint to update the weights of normalization layers for better generalization. We conduct extensive experiments on various datasets using several open-sourced LLMs. Our method demonstrates significant improvements in both weight-only quantization and joint quantization of weights and activations, surpassing existing PTQ methods. On GLM-130B and OPT-66B, our method even achieves the same level of accuracy at 2-bit quantization as their float ones. Our simple and effective approach makes it more practical for real-world applications.

EAAI Journal 2024 Journal Article

SOFT: Self-supervised sparse Optical Flow Transformer for video stabilization via quaternion

• Naiyao Wang
• Changdong Zhou
• Rongfeng Zhu
• Bo Zhang
• Ye Wang
• Hongbo Liu

Video stabilization is crucial for video representation learning, which suffers from the challenges such as the perception of unstable vision, the stripping and cognition of target motion features in complex scenes, the correction of the jittery camera systems trails. In this paper, we propose a Self-supervised sparse Optical Flow Transformer (SOFT) model, consisting of a self-supervised contrastive learning transformer network, a sparse optical flow perception network and a multimodal cognitive fusion network. The SOFT model takes advantage of optical flow to estimate motion. The sparse optical flow perception network perceiving partially sparse optical flow containing motion features. This serves as the input to the self-supervised contrastive learning transformer network for generating sparse optical flow features, which are fed into the multimodal cognitive fusion network together with the real and virtual camera pose for video frame warping. Experimental comparisons with state-of-the-art models on 4 metrics demonstrate the effectiveness of the SOFT model. It achieves the best performance with an average Stability of 0. 869 and average Distortion of 0. 993 across 6 categories videos, which shows that the SOFT model can effectively perceive the motion in the video and smooth the jitter track of videos.

NeurIPS Conference 2024 Conference Paper

Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy

• Hancheng Ye
• Jiakang Yuan
• Renqiu Xia
• Xiangchao Yan
• Tao Chen
• Junchi Yan
• Botian Shi
• Bo Zhang

Diffusion models have recently achieved great success in the synthesis of high-quality images and videos. However, the existing denoising techniques in diffusion models are commonly based on step-by-step noise predictions, which suffers from high computation cost, resulting in a prohibitive latency for interactive applications. In this paper, we propose AdaptiveDiffusion to relieve this bottleneck by adaptively reducing the noise prediction steps during the denoising process. Our method considers the potential of skipping as many noise prediction steps as possible while keeping the final denoised results identical to the original full-step ones. Specifically, the skipping strategy is guided by the third-order latent difference that indicates the stability between timesteps during the denoising process, which benefits the reusing of previous noise prediction results. Extensive experiments on image and video diffusion models demonstrate that our method can significantly speed up the denoising process while generating identical results to the original process, achieving up to an average 2-5x speedup without quality degradation. The code is available at https: //github. com/UniModal4Reasoning/AdaptiveDiffusion

NeurIPS Conference 2024 Conference Paper

ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Driving

• Tao Ma
• Hongbin Zhou
• Qiusheng Huang
• Xuemeng Yang
• Jianfei Guo
• Bo Zhang
• Min Dou
• Yu Qiao

Offboard perception aims to automatically generate high-quality 3D labels for autonomous driving (AD) scenes. Existing offboard methods focus on 3D object detection with closed-set taxonomy and fail to match human-level recognition capability on the rapidly evolving perception tasks. Due to heavy reliance on human labels and the prevalence of data imbalance and sparsity, a unified framework for offboard auto-labeling various elements in AD scenes that meets the distinct needs of perception tasks is not being fully explored. In this paper, we propose a novel multi-modal Zero-shot Offboard Panoptic Perception (ZOPP) framework for autonomous driving scenes. ZOPP integrates the powerful zero-shot recognition capabilities of vision foundation models and 3D representations derived from point clouds. To the best of our knowledge, ZOPP represents a pioneering effort in the domain of multi-modal panoptic perception and auto labeling for autonomous driving scenes. We conduct comprehensive empirical studies and evaluations on Waymo open dataset to validate the proposed ZOPP on various perception tasks. To further explore the usability and extensibility of our proposed ZOPP, we also conduct experiments in downstream applications. The results further demonstrate the great potential of our ZOPP for real-world scenarios. The source code will be released at \url{https: //github. com/PJLab-ADG/ZOPP}.

NeurIPS Conference 2023 Conference Paper

AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset

• Jiakang Yuan
• Bo Zhang
• Xiangchao Yan
• Botian Shi
• Tao Chen
• Yikang Li
• Yu Qiao

It is a long-term vision for Autonomous Driving (AD) community that the perception models can learn from a large-scale point cloud dataset, to obtain unified representations that can achieve promising results on different tasks or benchmarks. Previous works mainly focus on the self-supervised pre-training pipeline, meaning that they perform the pre-training and fine-tuning on the same benchmark, which is difficult to attain the performance scalability and cross-dataset application for the pre-training checkpoint. In this paper, for the first time, we are committed to building a large-scale pre-training point-cloud dataset with diverse data distribution, and meanwhile learning generalizable representations from such a diverse pre-training dataset. We formulate the point-cloud pre-training task as a semi-supervised problem, which leverages the few-shot labeled and massive unlabeled point-cloud data to generate the unified backbone representations that can be directly applied to many baseline models and benchmarks, decoupling the AD-related pre-training process and downstream fine-tuning task. During the period of backbone pre-training, by enhancing the scene- and instance-level distribution diversity and exploiting the backbone's ability to learn from unknown instances, we achieve significant performance gains on a series of downstream perception benchmarks including Waymo, nuScenes, and KITTI, under different baseline models like PV-RCNN++, SECOND, CenterPoint.

EAAI Journal 2023 Journal Article

EPT: A data-driven transformer model for earthquake prediction

• Bo Zhang
• Ziang Hu
• Pin Wu
• Haiwang Huang
• Jiansheng Xiang

The leading causes of earthquakes are crustal movements, plate movements, and collisions. In recent years, many researchers in earthquake prediction have been predicting earthquakes from historical seismic data in local areas. This approach ignores the underlying internal patterns of crustal motion, plate movement, and collisions. This paper proposes a purely data-driven deep learning model called EPT. The model uses gated feature extraction blocks (GFEB) to mine potential crustal motion and plate movement patterns from global historical seismic catalog data. It uses them to aid mainshock prediction in each local, provincial region. Experiments show that this approach improves model prediction accuracy by up to 50 percent. We also use multi-headed self-attention for the first time to capture long-term dependencies within regional time series, highlighting links between focal features and compensating for the difficulty of focusing on longer-term information in long-term time series with long short-term memory networks (LSTM). In addition, we also use the gradient harmonization mechanism classification (GHMC) loss function for the first time in earthquake prediction, effectively addressing the problem of uneven data distribution across different earthquake magnitude ranges. Finally, we validated the effectiveness of the EPT model in five provincial datasets in mainland China, and the experimental results all achieved an accuracy of over 90 percent.

ICML Conference 2023 Conference Paper

LipsNet: A Smooth and Robust Neural Network with Adaptive Lipschitz Constant for High Accuracy Optimal Control

• Xujie Song
• Jingliang Duan
• Wenxuan Wang 0004
• Shengbo Eben Li
• Chen Chen 0068
• Bo Cheng 0003
• Bo Zhang
• Junqing Wei

Deep reinforcement learning (RL) is a powerful approach for solving optimal control problems. However, RL-trained policies often suffer from the action fluctuation problem, where the consecutive actions significantly differ despite only slight state variations. This problem results in mechanical components’ wear and tear and poses safety hazards. The action fluctuation is caused by the high Lipschitz constant of actor networks. To address this problem, we propose a neural network named LipsNet. We propose the Multi-dimensional Gradient Normalization (MGN) method, to constrain the Lipschitz constant of networks with multi-dimensional input and output. Benefiting from MGN, LipsNet achieves Lipschitz continuity, allowing smooth actions while preserving control performance by adjusting Lipschitz constant. LipsNet addresses the action fluctuation problem at network level rather than algorithm level, which can serve as actor networks in most RL algorithms, making it more flexible and user-friendly than previous works. Experiments demonstrate that LipsNet has good landscape smoothness and noise robustness, resulting in significantly smoother action compared to the Multilayer Perceptron.

AAAI Conference 2023 Conference Paper

OPT-GAN: A Broad-Spectrum Global Optimizer for Black-Box Problems by Learning Distribution

• Minfang Lu
• Shuai Ning
• Shuangrong Liu
• Fengyang Sun
• Bo Zhang
• Bo Yang
• Lin Wang

Black-box optimization (BBO) algorithms are concerned with finding the best solutions for problems with missing analytical details. Most classical methods for such problems are based on strong and fixed a priori assumptions, such as Gaussianity. However, the complex real-world problems, especially when the global optimum is desired, could be very far from the a priori assumptions because of their diversities, causing unexpected obstacles. In this study, we propose a generative adversarial net-based broad-spectrum global optimizer (OPT-GAN) which estimates the distribution of optimum gradually, with strategies to balance exploration-exploitation trade-off. It has potential to better adapt to the regularity and structure of diversified landscapes than other methods with fixed prior, e.g., Gaussian assumption or separability. Experiments on diverse BBO benchmarks and high dimensional real world applications exhibit that OPT-GAN outperforms other traditional and neural net-based BBO algorithms. The code and Appendix are available at https://github.com/NBICLAB/OPT-GAN

JBHI Journal 2022 Journal Article

A Scalable Graph-Based Framework for Multi-Organ Histology Image Classification

• Yu Bai
• Yue Mi
• Yihan Su
• Bo Zhang
• Zheng Zhang
• Jingyun Wu
• Haiwen Huang
• Yongping Xiong

Graph-based approaches are successful for histology image classification tasks but still face many challenges, such as: 1) the lack of nuclei-level labels and the significant variations between histology images make it extremely difficult to extract discriminative high-level nuclei features like nuclei type, texture and micro-environment; 2) graph-based approaches cannot handle large-scale cell graph nodes typically contained in histology images; and 3) graph neural networks (GNNs) struggle to learn the long-range dependency of cell graphs. To address the above challenges, we propose a scalable graph-based framework for multi-organ histology image classification. We develop a two-step masked nuclei patches supervised training approach to extract discriminative high-level nuclei features for histology images without nuclei-level labels. Additionally, we introduce a nuclei sampling strategy to make our graph-based framework scalable for large-scale cell graphs. Furthermore, we propose H ier A rchical T ransformer Graph Neural Net work (HAT-Net+) for cell graph classi- fications. HAT-Net+ adopts Transformer to model the long-range dependency of cell graphs and a parameter-free approach to adaptively fuse different hierarchical graph representations of each layer. We achieved the state-of-the-art results on four public histology image classification datasets: CRC dataset (100%), Extended CRC dataset (98%), UZH dataset (96. 9%) and BACH dataset (88%). Unlike other methods, our approach can be used in various histology image classification tasks, even for images without nuclei-level labels, indicating its potential in cancer diagnosis. The code is available at https://github.com/suyouooooo/HAT-Net.

YNIMG Journal 2022 Journal Article

Distinct networks coupled with parietal cortex for spatial representations inside and outside the visual field

• Bo Zhang
• Fan Wang
• Qi Zhang
• Yuji Naya

Our mental representation of egocentric space is influenced by the disproportionate sensory perception of the body. Previous studies have focused on the neural architecture for egocentric representations within the visual field. However, the space representation underlying the body is still unclear. To address this problem, we applied both functional Magnitude Resonance Imaging (fMRI) and Magnetoencephalography (MEG) to a spatial-memory paradigm by using a virtual environment in which human participants remembered a target location left, right, or back relative to their own body. Both experiments showed larger involvement of the frontoparietal network in representing a retrieved target on the left/right side than on the back. Conversely, the medial temporal lobe (MTL)-parietal network was more involved in retrieving a target behind the participants. The MEG data showed an earlier activation of the MTL-parietal network than that of the frontoparietal network during retrieval of a target location. These findings suggest that the parietal cortex may represent the entire space around the self-body by coordinating two distinct brain networks.

EAAI Journal 2022 Journal Article

Human trajectory forecasting using a flow-based generative model

• Bo Zhang
• Tao Wang
• Changdong Zhou
• Nicola Conci
• Hongbo Liu

In this article, we present a flow-based framework for multi-modal trajectory prediction, which is able to provide an accurate and explicit inference of the latent representations on trajectory data. Differently from other typical generative models (such as GAN, VAE, etc.), the flow-based models aim at learning data distribution explicitly through an invertible network, which can convert a complicated distribution into a tractable form via invertible transformations. The whole framework is built upon the standard encoder–decoder architecture, where the LSTM is exploited as the fundamental block to capture the temporal structure of a trajectory. As a core module, we incorporate an invertible network that can learn the multi-modal distributions of trajectory data and further generate plausible future paths by sampling tricks from the standard Gaussian distribution. Extensive experiments carried out on synthetic and realistic datasets demonstrate the effectiveness of the proposed approach, and show the advantages as compared to the GAN-based and the VAE-based prediction frameworks.

CLeaR Conference 2022 Conference Paper

Some Reflections on Drawing Causal Inference using Textual Data: Parallels Between Human Subjects and Organized Texts

• Bo Zhang
• Jiayao Zhang

We examine the role of textual data as study units when conducting causal inference by drawing parallels between human subjects and organized texts. We elaborate on key causal concepts and principles, and expose some ambiguity and sometimes fallacies. To facilitate better framing a causal query, we discuss two strategies: (i) shifting from immutable traits to perceptions of them, and (ii) shifting from some abstract concept/property to its constituent parts, i. e. , a constructivist perspective of an abstract concept. We hope this article would raise the awareness of the importance of articulating and clarifying fundamental concepts before delving into developing methodologies when drawing causal inference using textual data.

ICRA Conference 2021 Conference Paper

Lywal: a Leg-Wheel Transformable Quadruped Robot with Picking up and Transport Functions

• Yongjiang Xue
• Xichen Yuan
• Yuhai Wang
• Yang Yang
• Siyu Lu
• Bo Zhang
• Juezhu Lai
• Jianming Wang

This paper introduces a leg-wheel transformable quadruped robot named Lywal which can switch to the leg-mode and the wheel-mode for locomotion, and the claw-mode for picking up and transport functions. First, the mechanical structure of Lywal is designed by using an innovative 2-DoF transformable mechanism. Second, the calculation of kinematics is analyzed in detail. Then, the switching-mode strategy and the mobile control strategies in different modes are designed. Finally, the prototype of Lywal is built. The properties of the mobile modes are analyzed, and the picking-up and transport functions of the claw-mode are verified through physical experiments.

JBHI Journal 2021 Journal Article

ML-Net: Multi-Channel Lightweight Network for Detecting Myocardial Infarction

• Yangjie Cao
• Tingting Wei
• Bo Zhang
• Nan Lin
• Joel J. P. C. Rodrigues
• Jie Li
• Di Zhang

Due to the complexity of myocardial infarction (MI) waveform, most traditional automatic diagnosis models rarely detect it, while those able to detect MI often require high computing and storage capacity, rendering them unsuitable for portable devices. Therefore, in order for convenient real-time MI detection, it is essential to design lightweight models suitable for resource-limited portable devices. This paper proposes a novel multi-channel lightweight model (ML-Net), that provides a new solution for portable detection devices with limited resources. In ML-Net, each electrocardiogram (ECG) lead is assigned an independent channel, ensuring data independence and preserve the ECG characteristics of different angles represented by different leads. Moreover, convolution kernels of heterogeneous sizes are utilized to achieve accurate classification with only a small amount of lead data. Extensive experiments over actual ECG data from the PTB diagnostic database are conducted to evaluate ML-Net. The results show that ML-Net outperforms comparable schemes in diagnosing MI, and it requires lower computational cost and less memory, so that portable devices can be more widely used in the field of Internet of Medical Things(IoMT).

NeurIPS Conference 2021 Conference Paper

Stability and Generalization of Bilevel Programming in Hyperparameter Optimization

• Fan Bao
• Guoqiang Wu
• Chongxuan Li
• Jun Zhu
• Bo Zhang

The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w. r. t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the classical cross-validation algorithm. Our results suggest that gradient-based algorithms can be better than cross-validation under certain conditions in a theoretical perspective. Furthermore, we prove that regularization terms in both the outer and inner levels can relieve the overfitting problem in gradient-based algorithms. In experiments on feature learning and data reweighting for noisy labels, we corroborate our theoretical findings.

EAAI Journal 2021 Journal Article

STENet: A hybrid spatio-temporal embedding network for human trajectory forecasting

• Bo Zhang
• Chengzhi Yuan
• Tao Wang
• Hongbo Liu

In this paper, we present a hybrid spatio-temporal embedding network (named as STENet) for human trajectory forecasting, which is built upon a GAN-based hierarchical framework. Differently from traditional approaches that only use LSTM for trajectory modeling, we exploit the 1D Convolutional Neural Network (1D-CNN) to embed position features at multiple temporal scales. Moreover, we propose a two-stage graph attention mechanism, which can better describe mutual interactions among pedestrians in the crowd. Additionally, group influences at every time step are taken into account as well. The overall framework is designed using a hierarchical manner, and trained using the Wasserstein distance. We carry out our experiments on the ETH and the UCY datasets. The corresponding results demonstrate the effectiveness of the proposed framework.

NeurIPS Conference 2021 Conference Paper

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

• Xiangxiang Chu
• Zhi Tian
• Yuqing Wang
• Bo Zhang
• Haibing Ren
• Xiaolin Wei
• Huaxia Xia
• Chunhua Shen

Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spatial attention and demonstrate that a carefully devised yet simple spatial attention mechanism performs favorably against the state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, Twins- PCPVT and Twins-SVT. Our proposed architectures are highly efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent performance on a wide range of visual tasks including image-level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks.

IJCAI Conference 2020 Conference Paper

An Attention-based Model for Conversion Rate Prediction with Delayed Feedback via Post-click Calibration

• Yumin Su
• Liang Zhang
• Quanyu Dai
• Bo Zhang
• Jinyao Yan
• Dan Wang
• Yongjun Bao
• Sulong Xu

Conversion rate (CVR) prediction is becoming increasingly important in the multi-billion dollar online display advertising industry. It has two major challenges: firstly, the scarce user history data is very complicated and non-linear; secondly, the time delay between the clicks and the corresponding conversions can be very large, e. g. , ranging from seconds to weeks. Existing models usually suffer from such scarce and delayed conversion behaviors. In this paper, we propose a novel deep learning framework to tackle the two challenges. Specifically, we extract the pre-trained embedding from impressions/clicks to assist in conversion models and propose an inner/self-attention mechanism to capture the fine-grained personalized product purchase interests from the sequential click data. Besides, to overcome the time-delay issue, we calibrate the delay model by learning dynamic hazard function with the abundant post-click data more in line with the real distribution. Empirical experiments with real-world user behavior data prove the effectiveness of the proposed method.

NeurIPS Conference 2020 Conference Paper

Bi-level Score Matching for Learning Energy-based Latent Variable Models

• Fan Bao
• Chongxuan Li
• Kun Xu
• Hang Su
• Jun Zhu
• Bo Zhang

Score matching (SM) provides a compelling approach to learn energy-based models (EBMs) by avoiding the calculation of partition function. However, it remains largely open to learn energy-based latent variable models (EBLVMs), except some special cases. This paper presents a bi-level score matching (BiSM) method to learn EBLVMs with general structures by reformulating SM as a bi-level optimization problem. The higher level introduces a variational posterior of the latent variables and optimizes a modified SM objective, and the lower level optimizes the variational posterior to fit the true posterior. To solve BiSM efficiently, we develop a stochastic optimization algorithm with gradient unrolling. Theoretically, we analyze the consistency of BiSM and the convergence of the stochastic algorithm. Empirically, we show the promise of BiSM in Gaussian restricted Boltzmann machines and highly nonstructural EBLVMs parameterized by deep convolutional neural networks. BiSM is comparable to the widely adopted contrastive divergence and SM methods when they are applicable; and can learn complex EBLVMs with intractable posteriors to generate natural images.

AAAI Conference 2020 Conference Paper

Dynamic Network Pruning with Interpretable Layerwise Channel Selection

• Yulong Wang
• Xiaolu Zhang
• Xiaolin Hu
• Bo Zhang
• Hang Su

Dynamic network pruning achieves runtime acceleration by dynamically determining the inference paths based on different inputs. However, previous methods directly generate continuous decision values for each weight channel, which cannot reflect a clear and interpretable pruning process. In this paper, we propose to explicitly model the discrete weight channel selections, which encourages more diverse weights utilization, and achieves more sparse runtime inference paths. Meanwhile, with the help of interpretable layerwise channel selections in the dynamic network, we can visualize the network decision paths explicitly for model interpretability. We observe that there are clear differences in the layerwise decisions between normal and adversarial examples. Therefore, we propose a novel adversarial example detection algorithm by discriminating the runtime decision features. Experiments show that our dynamic network achieves higher prediction accuracy under the similar computing budgets on CIFAR10 and ImageNet datasets compared to traditional static pruning methods and other dynamic pruning approaches. The proposed adversarial detection algorithm can significantly improve the state-of-the-art detection rate across multiple attacks, which provides an opportunity to build an interpretable and robust model.

IJCAI Conference 2020 Conference Paper

Internal and Contextual Attention Network for Cold-start Multi-channel Matching in Recommendation

• Ruobing Xie
• Zhijie Qiu
• Jun Rao
• Yi Liu
• Bo Zhang
• Leyu Lin

Real-world integrated personalized recommendation systems usually deal with millions of heterogeneous items. It is extremely challenging to conduct full corpus retrieval with complicated models due to the tremendous computation costs. Hence, most large-scale recommendation systems consist of two modules: a multi-channel matching module to efficiently retrieve a small subset of candidates, and a ranking module for precise personalized recommendation. However, multi-channel matching usually suffers from cold-start problems when adding new channels or new data sources. To solve this issue, we propose a novel Internal and contextual attention network (ICAN), which highlights channel-specific contextual information and feature field interactions between multiple channels. In experiments, we conduct both offline and online evaluations with case studies on a real-world integrated recommendation system. The significant improvements confirm the effectiveness and robustness of ICAN, especially for cold-start channels. Currently, ICAN has been deployed on WeChat Top Stories used by millions of users. The source code can be obtained from https: //github. com/zhijieqiu/ICAN.

ECAI Conference 2020 Conference Paper

Joint Modeling of Local and Global Behavior Dynamics for Session-Based Recommendation

• Yong Xu 0007
• Jiahui Chen
• Chao Huang 0001
• Bo Zhang
• Hao Xing
• Peng Dai 0001
• Liefeng Bo

Session-based recommendation is critical in modern recommender systems, which aims to predict the next interested item given anonymous behavior sequences of users. While prior works have made efforts to addressing the session-based recommendation problem, two significant limitations exist: i) They ignore the fact that items may be correlated with other across different session units; ii) existing solutions are also limited in their assumption of rigidly ordered pattern over intra-session item transition, which may not be true in practice. To address these above limitations, we propose a Local-Global Session-based Recommendation framework–LGSR which generalizes the modeling of behavior dynamics from two perspectives: we first design a cross-session item dependency encoder to learn the inter-session item relation structures from a global perspective. Additionally, a dual-stage attentive aggregation module is developed to capture local item transition dynamics, without the restriction of rigid sequential process for jointly modeling user’s current interest and intra-session purpose. With the exploration of both complex intra- and inter-session interest transitional regularities, our LGSR model enables the representation learning of user behavior dynamics via jointly mapping local and global signals into the same latent space. The experimental results on two real-world datasets demonstrate the superiority of the proposed LGSR framework over state-of-the-art methods.

AAAI Conference 2020 Conference Paper

Pruning from Scratch

• Yulong Wang
• Xiaolu Zhang
• Lingxi Xie
• Jun Zhou
• Hang Su
• Bo Zhang
• Xiaolin Hu

Network pruning is an important research field aiming at reducing computational costs of neural networks. Conventional approaches follow a fixed paradigm which first trains a large and redundant network, and then determines which units (e. g. , channels) are less important and thus can be removed. In this work, we find that pre-training an over-parameterized model is not necessary for obtaining the target pruned structure. In fact, a fully-trained over-parameterized model will reduce the search space for the pruned structure. We empirically show that more diverse pruned structures can be directly pruned from randomly initialized weights, including potential models with better performance. Therefore, we propose a novel network pruning pipeline which allows pruning from scratch with little training overhead. In the experiments for compressing classification models on CIFAR10 and ImageNet datasets, our approach not only greatly reduces the pre-training burden of traditional pruning methods, but also achieves similar or even higher accuracy under the same computation budgets. Our results facilitate the community to rethink the effectiveness of existing techniques used for network pruning.

NeurIPS Conference 2019 Conference Paper

Multi-objects Generation with Amortized Structural Regularization

• Taufik Xu
• Chongxuan Li
• Jun Zhu
• Bo Zhang

Deep generative models (DGMs) have shown promise in image generation. However, most of the existing methods learn a model by simply optimizing a divergence between the marginal distributions of the model and the data, and often fail to capture rich structures, such as attributes of objects and their relationships, in an image. Human knowledge is a crucial element to the success of DGMs to infer these structures, especially in unsupervised learning. In this paper, we propose amortized structural regularization (ASR), which adopts posterior regularization (PR) to embed human knowledge into DGMs via a set of structural constraints. We derive a lower bound of the regularized log-likelihood in PR and adopt the amortized inference technique to jointly optimize the generative model and an auxiliary recognition model for inference efficiently. Empirical results show that ASR outperforms the DGM baselines in terms of inference performance and sample quality.

JBHI Journal 2019 Journal Article

Multitask Cascade Convolution Neural Networks for Automatic Thyroid Nodule Detection and Recognition

• Wenfeng Song
• Shuai Li
• Ji Liu
• Hong Qin
• Bo Zhang
• Shuyang Zhang
• Aimin Hao

Thyroid ultrasonography is a widely used clinical technique for nodule diagnosis in thyroid regions. However, it remains difficult to detect and recognize the nodules due to low contrast, high noise, and diverse appearance of nodules. In today's clinical practice, senior doctors could pinpoint nodules by analyzing global context features, local geometry structure, and intensity changes, which would require rich clinical experience accumulated from hundreds and thousands of nodule case studies. To alleviate doctors’ tremendous labor in the diagnosis procedure, we advocate a machine learning approach to the detection and recognition tasks in this paper. In particular, we develop a multitask cascade convolution neural network (MC-CNN) framework to exploit the context information of thyroid nodules. It may be noted that our framework is built upon a large number of clinically confirmed thyroid ultrasound images with accurate and detailed ground truth labels. Other key advantages of our framework result from a multitask cascade architecture, two stages of carefully designed deep convolution networks in order to detect and recognize thyroid nodules in a pyramidal fashion, and capturing various intrinsic features in a global-to-local way. Within our framework, the potential regions of interest after initial detection are further fed to the spatial pyramid augmented CNNs to embed multiscale discriminative information for fine-grained thyroid recognition. Experimental results on 4309 clinical ultrasound images have indicated that our MC-CNN is accurate and effective for both thyroid nodules detection and recognition. For the correct diagnosis rate of malignant and benign thyroid nodules, its mean Average Precision (mAP) performance can achieve up to $\text{98. 2}\%$ accuracy, which outperforms the common CNNs by $\text{5}\%$ on average. In addition, we conduct rigorous user studies to confirm that our MC-CNN outperforms experienced doctors, yet only consuming roughly $\text{2}\%$ ( $1/48$ ) of doctors’ examination time on average. Therefore, the accuracy and efficiency of our new method exhibit its great potential in clinical applications.

AAAI Conference 2018 Conference Paper

Collaborative Filtering With User-Item Co-Autoregressive Models

• Chao Du
• Chongxuan Li
• Yin Zheng
• Jun Zhu
• Bo Zhang

Deep neural networks have shown promise in collaborative filtering (CF). However, existing neural approaches are either user-based or item-based, which cannot leverage all the underlying information explicitly. We propose CF-UIcA, a neural co-autoregressive model for CF tasks, which exploits the structural correlation in the domains of both users and items. The co-autoregression allows extra desired properties to be incorporated for different tasks. Furthermore, we develop an efficient stochastic learning algorithm to handle large scale datasets. We evaluate CF-UIcA on two popular benchmarks: MovieLens 1M and Netflix, and achieve state-of-the-art performance in both rating prediction and top-N recommendation tasks, which demonstrates the effectiveness of CF-UIcA.

NeurIPS Conference 2018 Conference Paper

DeepExposure: Learning to Expose Photos with Asynchronously Reinforced Adversarial Learning

• Runsheng Yu
• Wenyu Liu
• Yasen Zhang
• Zhi Qu
• Deli Zhao
• Bo Zhang

The accurate exposure is the key of capturing high-quality photos in computational photography, especially for mobile phones that are limited by sizes of camera modules. Inspired by luminosity masks usually applied by professional photographers, in this paper, we develop a novel algorithm for learning local exposures with deep reinforcement adversarial learning. To be specific, we segment an image into sub-images that can reflect variations of dynamic range exposures according to raw low-level features. Based on these sub-images, a local exposure for each sub-image is automatically learned by virtue of policy network sequentially while the reward of learning is globally designed for striking a balance of overall exposures. The aesthetic evaluation function is approximated by discriminator in generative adversarial networks. The reinforcement learning and the adversarial learning are trained collaboratively by asynchronous deterministic policy gradient and generative loss approximation. To further simply the algorithmic architecture, we also prove the feasibility of leveraging the discriminator as the value function. Further more, we employ each local exposure to retouch the raw input image respectively, thus delivering multiple retouched images under different exposures which are fused with exposure blending. The extensive experiments verify that our algorithms are superior to state-of-the-art methods in terms of quantitative accuracy and visual illustration.

NeurIPS Conference 2018 Conference Paper

Graphical Generative Adversarial Networks

• Chongxuan Li
• Max Welling
• Jun Zhu
• Bo Zhang

We propose Graphical Generative Adversarial Networks (Graphical-GAN) to model structured data. Graphical-GAN conjoins the power of Bayesian networks on compactly representing the dependency structures among random variables and that of generative adversarial networks on learning expressive dependency functions. We introduce a structured recognition model to infer the posterior distribution of latent variables given observations. We generalize the Expectation Propagation (EP) algorithm to learn the generative model and recognition model jointly. Finally, we present two important instances of Graphical-GAN, i. e. Gaussian Mixture GAN (GMGAN) and State Space GAN (SSGAN), which can successfully learn the discrete and temporal structures on visual datasets, respectively.

NeurIPS Conference 2018 Conference Paper

Semi-crowdsourced Clustering with Deep Generative Models

• Yucen Luo
• Tian Tian
• Jiaxin Shi
• Jun Zhu
• Bo Zhang

We consider the semi-supervised clustering problem where crowdsourcing provides noisy information about the pairwise comparisons on a small subset of data, i. e. , whether a sample pair is in the same cluster. We propose a new approach that includes a deep generative model (DGM) to characterize low-level features of the data, and a statistical relational model for noisy pairwise annotations on its subset. The two parts share the latent variables. To make the model automatically trade-off between its complexity and fitting data, we also develop its fully Bayesian variant. The challenge of inference is addressed by fast (natural-gradient) stochastic variational inference algorithms, where we effectively combine variational message passing for the relational part and amortized learning of the DGM under a unified framework. Empirical results on synthetic and real-world datasets show that our model outperforms previous crowdsourced clustering methods.

IJCAI Conference 2017 Conference Paper

Forecast the Plausible Paths in Crowd Scenes

• Hang Su
• Jun Zhu
• Yinpeng Dong
• Bo Zhang

Forecasting the future plausible paths of pedestrians in crowd scenes is of wide applications, but it still remains as a challenging task due to the complexities and uncertainties of crowd motions. To address these issues, we propose to explore the inherent crowd dynamics via a social-aware recurrent Gaussian process model, which facilitates the path prediction by taking advantages of the interplay between the rich prior knowledge and motion uncertainties. Specifically, we derive a social-aware LSTM to explore the crowd dynamic, resulting in a hidden feature embedding the rich prior in massive data. Afterwards, we integrate the descriptor into deep Gaussian processes with motion uncertainties appropriately harnessed. Crowd motion forecasting is implemented by regressing relative motion against the current positions, yielding the predicted paths based on a functional object associated with a distribution. Extensive experiments on public datasets demonstrate that our method obtains the state-of-the-art performance in both structured and unstructured scenes by exploring the complex and uncertain motion patterns, even if the occlusion is serious or the observed trajectories are noisy.

IJCAI Conference 2017 Conference Paper

Semi-supervised Max-margin Topic Model with Manifold Posterior Regularization

• Wenbo Hu
• Jun Zhu
• Hang Su
• Jingwei Zhuo
• Bo Zhang

Supervised topic models leverage label information to learn discriminative latent topic representations. As collecting a fully labeled dataset is often time-consuming, semi-supervised learning is of high interest. In this paper, we present an effective semi-supervised max-margin topic model by naturally introducing manifold posterior regularization to a regularized Bayesian topic model, named LapMedLDA. The model jointly learns latent topics and a related classifier with only a small fraction of labeled documents. To perform the approximate inference, we derive an efficient stochastic gradient MCMC method. Unlike the previous semi-supervised topic models, our model adopts a tight coupling between the generative topic model and the discriminative classifier. Extensive experiments demonstrate that such tight coupling brings significant benefits in quantitative and qualitative performance.

NeurIPS Conference 2017 Conference Paper

Triple Generative Adversarial Nets

• Chongxuan Li
• Taufik Xu
• Jun Zhu
• Bo Zhang

Generative Adversarial Nets (GANs) have shown promise in image generation and semi-supervised learning (SSL). However, existing GANs in SSL have two problems: (1) the generator and the discriminator (i. e. the classifier) may not be optimal at the same time; and (2) the generator cannot control the semantics of the generated samples. The problems essentially arise from the two-player formulation, where a single discriminator shares incompatible roles of identifying fake samples and predicting labels and it only estimates the data without considering the labels. To address the problems, we present triple generative adversarial net (Triple-GAN), which consists of three players---a generator, a discriminator and a classifier. The generator and the classifier characterize the conditional distributions between images and labels, and the discriminator solely focuses on identifying fake image-label pairs. We design compatible utilities to ensure that the distributions characterized by the classifier and the generator both converge to the data distribution. Our results on various datasets demonstrate that Triple-GAN as a unified model can simultaneously (1) achieve the state-of-the-art classification results among deep generative models, and (2) disentangle the classes and styles of the input and transfer smoothly in the data space via interpolation in the latent space class-conditionally.

IJCAI Conference 2016 Conference Paper

Crowd Scene Understanding with Coherent Recurrent Neural Networks

• Hang Su
• Yinpeng Dong
• Jun Zhu
• Haibin Ling
• Bo Zhang

Exploring crowd dynamics is essential in understanding crowd scenes, which still remains as a challenging task due to the nonlinear characteristics and coherent spatio-temporal motion patterns in crowd behaviors. To address these issues, we present a Coherent Long Short Term Memory (cLSTM) network to capture the nonlinear crowd dynamics by learning an informative representation of crowd motions, which facilitates the critical tasks in crowd scene analysis. By describing the crowd motion patterns with a cloud of keypoint tracklets, we explore the nonlinear crowd dynamics embedded in the tracklets with a stacked LSTM model, which is further improved to capture the collective properties by introducing a coherent regularization term; and finally, we adopt an unsupervised encoder-decoder framework to learn a hidden feature for each input tracklet that embeds its inherent dynamics. With the learnt features properly harnessed, crowd scene understanding is conducted effectively in predicting the future paths of agents, estimating group states, and classifying crowd events. Extensive experiments on hundreds of public crowd videos demonstrate that our method is state-of-the-art performance by exploring the coherent spatio-temporal structures in crowd behaviors.

AAAI Conference 2016 Conference Paper

Discriminative Nonparametric Latent Feature Relational Models with Data Augmentation

• Bei Chen
• Ning Chen
• Jun Zhu
• Jiaming Song
• Bo Zhang

We present a discriminative nonparametric latent feature relational model (LFRM) for link prediction to automatically infer the dimensionality of latent features. Under the generic RegBayes (regularized Bayesian inference) framework, we handily incorporate the prediction loss with probabilistic inference of a Bayesian model; set distinct regularization parameters for different types of links to handle the imbalance issue in real networks; and unify the analysis of both the smooth logistic log-loss and the piecewise linear hinge loss. For the nonconjugate posterior inference, we present a simple Gibbs sampler via data augmentation, without making restricting assumptions as done in variational methods. We further develop an approximate sampler using stochastic gradient Langevin dynamics to handle large networks with hundreds of thousands of entities and millions of links, orders of magnitude larger than what existing LFRM models can process. Extensive studies on various real networks show promising performance.

AAAI Conference 2016 Conference Paper

Jointly Modeling Topics and Intents with Global Order Structure

• Bei Chen
• Jun Zhu
• Nan Yang
• Tian Tian
• Ming Zhou
• Bo Zhang

Modeling document structure is of great importance for discourse analysis and related applications. The goal of this research is to capture the document intent structure by modeling documents as a mixture of topic words and rhetorical words. While the topics are relatively unchanged through one document, the rhetorical functions of sentences usually change following certain orders in discourse. We propose GMM-LDA, a topic modeling based Bayesian unsupervised model, to analyze the document intent structure cooperated with order information. Our model is flexible that has the ability to combine the annotations and do supervised learning. Additionally, entropic regularization can be introduced to model the significant divergence between topics and intents. We perform experiments in both unsupervised and supervised settings, results show the superiority of our model over several state-of-the-art baselines.

IJCAI Conference 2015 Conference Paper

Adaptive Dropout Rates for Learning with Corrupted Features

• Jingwei Zhuo
• Jun Zhu
• Bo Zhang

Feature noising is an effective mechanism on reducing the risk of overfitting. To avoid an explosive searching space, existing work typically assumes that all features share a single noise level, which is often cross-validated. In this paper, we present a Bayesian feature noising model that flexibly allows for dimension-specific or group-specific noise levels, and we derive a learning algorithm that adaptively updates these noise levels. Our adaptive rule is simple and interpretable, by drawing a direct connection to the fitness of each individual feature or feature group. Empirical results on various datasets demonstrate the effectiveness on avoiding extensive tuning and sometimes improving the performance due to its flexibility.

TIST Journal 2015 Journal Article

An Event-Driven QoI-Aware Participatory Sensing Framework with Energy and Budget Constraints

• Bo Zhang
• Zheng Song
• Chi Harold Liu
• Jian Ma
• Wendong Wang

Participatory sensing systems can be used for concurrent event monitoring applications, like noise levels, fire, and pollutant concentrations. However, they are facing new challenges as to how to accurately detect the exact boundaries of these events, and further, to select the most appropriate participants to collect the sensing data. On the one hand, participants’ handheld smart devices are constrained with different energy conditions and sensing capabilities, and they move around with uncontrollable mobility patterns in their daily life. On the other hand, these sensing tasks are within time-varying quality-of-information (QoI) requirements and budget to afford the users’ incentive expectations. Toward this end, this article proposes an event-driven QoI-aware participatory sensing framework with energy and budget constraints. The main method of this framework is event boundary detection. For the former, a two-step heuristic solution is proposed where the coarse-grained detection step finds its approximation and the fine-grained detection step identifies the exact location. Participants are selected by explicitly considering their mobility pattern, required QoI of multiple tasks, and users’ incentive requirements, under the constraint of an aggregated task budget. Extensive experimental results, based on a real trace in Beijing, show the effectiveness and robustness of our approach, while comparing with existing schemes.

NeurIPS Conference 2015 Conference Paper

Convolutional Neural Networks with Intra-Layer Recurrent Connections for Scene Labeling

• Ming Liang
• Xiaolin Hu
• Bo Zhang

Scene labeling is a challenging computer vision task. It requires the use of both local discriminative features and global context information. We adopt a deep recurrent convolutional neural network (RCNN) for this task, which is originally proposed for object recognition. Different from traditional convolutional neural networks (CNN), this model has intra-layer recurrent connections in the convolutional layers. Therefore each convolutional layer becomes a two-dimensional recurrent neural network. The units receive constant feed-forward inputs from the previous layer and recurrent inputs from their neighborhoods. While recurrent iterations proceed, the region of context captured by each unit expands. In this way, feature extraction and context modulation are seamlessly integrated, which is different from typical methods that entail separate modules for the two steps. To further utilize the context, a multi-scale RCNN is proposed. Over two benchmark datasets, Standford Background and Sift Flow, the model outperforms many state-of-the-art models in accuracy and efficiency.

NeurIPS Conference 2015 Conference Paper

Max-Margin Deep Generative Models

• Chongxuan Li
• Jun Zhu
• Tianlin Shi
• Bo Zhang

Deep generative models (DGMs) are effective on learning multilayered representations of complex data and performing inference of input data by exploring the generative ability. However, little work has been done on examining or empowering the discriminative ability of DGMs on making accurate predictions. This paper presents max-margin deep generative models (mmDGMs), which explore the strongly discriminative principle of max-margin learning to improve the discriminative power of DGMs, while retaining the generative capability. We develop an efficient doubly stochastic subgradient algorithm for the piecewise linear objective. Empirical results on MNIST and SVHN datasets demonstrate that (1) max-margin learning can significantly improve the prediction performance of DGMs and meanwhile retain the generative ability; and (2) mmDGMs are competitive to the state-of-the-art fully discriminative networks by employing deep convolutional neural networks (CNNs) as both recognition and generative models.

NeurIPS Conference 2014 Conference Paper

Distributed Bayesian Posterior Sampling via Moment Sharing

• Minjie Xu
• Balaji Lakshminarayanan
• Yee Whye Teh
• Jun Zhu
• Bo Zhang

We propose a distributed Markov chain Monte Carlo (MCMC) inference algorithm for large scale Bayesian posterior simulation. We assume that the dataset is partitioned and stored across nodes of a cluster. Our procedure involves an independent MCMC posterior sampler at each node based on its local partition of the data. Moment statistics of the local posteriors are collected from each sampler and propagated across the cluster using expectation propagation message passing with low communication costs. The moment sharing scheme improves posterior estimation quality by enforcing agreement among the samplers. We demonstrate the speed and inference quality of our method with empirical studies on Bayesian logistic regression and sparse linear regression with a spike-and-slab prior.

AAAI Conference 2014 Conference Paper

Dropout Training for Support Vector Machines

• Ning Chen
• Jun Zhu
• Jianfei Chen
• Bo Zhang

Dropout and other feature noising schemes have shown promising results in controlling over-fitting by artificially corrupting the training data. Though extensive theoretical and empirical studies have been performed for generalized linear models, little work has been done for support vector machines (SVMs), one of the most successful approaches for supervised learning. This paper presents dropout training for linear SVMs. To deal with the intractable expectation of the non-smooth hinge loss under corrupting distributions, we develop an iteratively re-weighted least square (IRLS) algorithm by exploring data augmentation techniques. Our algorithm iteratively minimizes the expectation of a re-weighted least square problem, where the re-weights have closedform solutions. The similar ideas are applied to develop a new IRLS algorithm for the expected logistic loss under corrupting distributions. Our algorithms offer insights on the connection and difference between the hinge loss and logistic loss in dropout training. Empirical results on several real datasets demonstrate the effectiveness of dropout training on significantly boosting the classification accuracy of linear SVMs.

JMLR Journal 2014 Journal Article

Gibbs Max-margin Topic Models with Data Augmentation

• Jun Zhu
• Ning Chen
• Hugh Perkins
• Bo Zhang

Max-margin learning is a powerful approach to building classifiers and structured output predictors. Recent work on max-margin supervised topic models has successfully integrated it with Bayesian topic models to discover discriminative latent semantic structures and make accurate predictions for unseen testing data. However, the resulting learning problems are usually hard to solve because of the non-smoothness of the margin loss. Existing approaches to building max-margin supervised topic models rely on an iterative procedure to solve multiple latent SVM subproblems with additional mean-field assumptions on the desired posterior distributions. This paper presents an alternative approach by defining a new max-margin loss. Namely, we present Gibbs max-margin supervised topic models, a latent variable Gibbs classifier to discover hidden topic representations for various tasks, including classification, regression and multi-task learning. Gibbs max- margin supervised topic models minimize an expected margin loss, which is an upper bound of the existing margin loss derived from an expected prediction rule. By introducing augmented variables and integrating out the Dirichlet variables analytically by conjugacy, we develop simple Gibbs sampling algorithms with no restrictive assumptions and no need to solve SVM subproblems. Furthermore, each step of the “augment-and-collapse" Gibbs sampling algorithms has an analytical conditional distribution, from which samples can be easily drawn. Experimental results on several medium-sized and large-scale data sets demonstrate significant improvements on time efficiency. The classification performance is also improved over competitors on binary, multi- class and multi-label classification tasks. [abs] [ pdf ][ bib ] &copy JMLR 2014. ( edit, beta )

AAAI Conference 2014 Conference Paper

Similarity-Preserving Binary Signature for Linear Subspaces

• Jianqiu Ji
• Jianmin Li
• Shuicheng Yan
• Qi Tian
• Bo Zhang

Linear subspace is an important representation for many kinds of real-world data in computer vision and pattern recognition, e. g. faces, motion videos, speeches. In this paper, first we define pairwise angular similarity and angular distance for linear subspaces. The angular distance satisfies non-negativity, identity of indiscernibles, symmetry and triangle inequality, and thus it is a metric. Then we propose a method to compress linear subspaces into compact similarity-preserving binary signatures, between which the normalized Hamming distance is an unbiased estimator of the angular distance. We provide a lower bound on the length of the binary signatures which suffices to guarantee uniform distancepreservation within a set of subspaces. Experiments on face recognition demonstrate the effectiveness of the binary signature in terms of recognition accuracy, speed and storage requirement. The results show that, compared with the exact method, the approximation with the binary signatures achieves an order of magnitude speedup, while requiring significantly smaller amount of storage space, yet it still accurately preserves the similarity, and achieves high recognition accuracy comparable to the exact method in face recognition.

IJCAI Conference 2013 Conference Paper

Generalized Relational Topic Models with Data Augmentation

• Ning Chen
• Jun Zhu
• Fei Xia
• Bo Zhang

Relational topic models have shown promise on analyzing document network structures and discovering latent topic representations. This paper presents three extensions: 1) unlike the common link likelihood with a diagonal weight matrix that allows the-same-topic interactions only, we generalize it to use a full weight matrix that captures all pairwise topic interactions and is applicable to asymmetric networks; 2) instead of doing standard Bayesian inference, we perform regularized Bayesian inference with a regularization parameter to deal with the imbalanced link structure issue in common real networks; and 3) instead of doing variational approximation with strict mean-field assumptions, we present a collapsed Gibbs sampling algorithm for the generalized relational topic models without making restricting assumptions. Experimental results demonstrate the significance of these extensions on improving the prediction performance, and the time efficiency can be dramatically improved with a simple fast approximation method.

NeurIPS Conference 2013 Conference Paper

Scalable Inference for Logistic-Normal Topic Models

• Jianfei Chen
• Jun Zhu
• Zi Wang
• Xun Zheng
• Bo Zhang

Logistic-normal topic models can effectively discover correlation structures among latent topics. However, their inference remains a challenge because of the non-conjugacy between the logistic-normal prior and multinomial topic mixing proportions. Existing algorithms either make restricting mean-field assumptions or are not scalable to large-scale applications. This paper presents a partially collapsed Gibbs sampling algorithm that approaches the provably correct distribution by exploring the ideas of data augmentation. To improve time efficiency, we further present a parallel implementation that can deal with large-scale applications and learn the correlation structures of thousands of topics from millions of documents. Extensive empirical results demonstrate the promise.

NeurIPS Conference 2012 Conference Paper

Nonparametric Max-Margin Matrix Factorization for Collaborative Prediction

• Minjie Xu
• Jun Zhu
• Bo Zhang

We present a probabilistic formulation of max-margin matrix factorization and build accordingly a nonparametric Bayesian model which automatically resolves the unknown number of latent factors. Our work demonstrates a successful example that integrates Bayesian nonparametrics and max-margin learning, which are conventionally two separate paradigms and enjoy complementary advantages. We develop an efcient variational algorithm for posterior inference, and our extensive empirical studies on large-scale MovieLens and EachMovie data sets appear to justify the aforementioned dual advantages.

NeurIPS Conference 2012 Conference Paper

Super-Bit Locality-Sensitive Hashing

• Jianqiu Ji
• Jianmin Li
• Shuicheng Yan
• Bo Zhang
• Qi Tian

Sign-random-projection locality-sensitive hashing (SRP-LSH) is a probabilistic dimension reduction method which provides an unbiased estimate of angular similarity, yet suffers from the large variance of its estimation. In this work, we propose the Super-Bit locality-sensitive hashing (SBLSH). It is easy to implement, which orthogonalizes the random projection vectors in batches, and it is theoretically guaranteed that SBLSH also provides an unbiased estimate of angular similarity, yet with a smaller variance when the angle to estimate is within $(0, \pi/2]$. The extensive experiments on real data well validate that given the same length of binary code, SBLSH may achieve significant mean squared error reduction in estimating pairwise angular similarity. Moreover, SBLSH shows the superiority over SRP-LSH in approximate nearest neighbor (ANN) retrieval experiments.

JMLR Journal 2008 Journal Article

Dynamic Hierarchical Markov Random Fields for Integrated Web Data Extraction

• Jun Zhu
• Zaiqing Nie
• Bo Zhang
• Ji-Rong Wen

Existing template-independent web data extraction approaches adopt highly ineffective decoupled strategies---attempting to do data record detection and attribute labeling in two separate phases. In this paper, we propose an integrated web data extraction paradigm with hierarchical models. The proposed model is called Dynamic Hierarchical Markov Random Fields (DHMRFs). DHMRFs take structural uncertainty into consideration and define a joint distribution of both model structure and class labels. The joint distribution is an exponential family distribution. As a conditional model, DHMRFs relax the independence assumption as made in directed models. Since exact inference is intractable, a variational method is developed to learn the model's parameters and to find the MAP model structure and label assignments. We apply DHMRFs to a real-world web data extraction task. Experimental results show that: (1) integrated web data extraction models can achieve significant improvements on both record detection and attribute labeling compared to decoupled models; (2) in diverse web data extraction DHMRFs can potentially address the blocky artifact issue which is suffered by fixed-structured hierarchical models. [abs] [ pdf ][ bib ] &copy JMLR 2008. ( edit, beta )

NeurIPS Conference 2008 Conference Paper

Partially Observed Maximum Entropy Discrimination Markov Networks

• Jun Zhu
• Eric Xing
• Bo Zhang

Learning graphical models with hidden variables can offer semantic insights to complex data and lead to salient structured predictors without relying on expensive, sometime unattainable fully annotated training data. While likelihood-based methods have been extensively explored, to our knowledge, learning structured prediction models with latent variables based on the max-margin principle remains largely an open problem. In this paper, we present a partially observed Maximum Entropy Discrimination Markov Network (PoMEN) model that attempts to combine the advantages of Bayesian and margin based paradigms for learning Markov networks from partially labeled data. PoMEN leads to an averaging prediction rule that resembles a Bayes predictor that is more robust to overfitting, but is also built on the desirable discriminative laws resemble those of the M$^3$N. We develop an EM-style algorithm utilizing existing convex optimization algorithms for M$^3$N as a subroutine. We demonstrate competent performance of PoMEN over existing methods on a real-world web data extraction task.

TCS Journal 2005 Journal Article

Approximation of Walrasian equilibrium in single-minded auctions

• Li-Sha Huang
• Minming Li
• Bo Zhang

We consider a social optimization model of pricing scheme in single-minded auctions, in cases where Walrasian equilibrium does not exist. We are interested in the maximization of the ratio, R, of happy bidders over all agents, in a feasible allocation-pricing scheme. We show NP-hardness of the optimization problem, establish lower and upper bounds of R, as well as develop greedy algorithms to approximate the optimal value of R.

ICRA Conference 1994 Conference Paper

A New Scheduling Model Based on Extended Petri Net - TREM Net

• Biqing Huang
• Bo Zhang

When modeling a scheduling problem by Petri net most researchers used the so-called pure net. The net can not deal with the allocations of time resources and the critical resources such as processors. These resources should be the elements of the net, so in the strict sense, the net is not a timed Petri net. In this paper, we present an extended Petri net model, a scheduling model with time resources ( called TREM net). TREM net can model both processors and times so it can easily treat the allocations of both simultaneously. We discuss in details the method of solving T-invariants of TREM net and the close relation between T-invariants and scheduling. The idea of combining AI technique with Petri net based scheduling model is also presented in the paper. >

IROS Conference 1994 Conference Paper

Control of robotic manipulators using a CMAC-based reinforcement learning system

• Mei Han
• Bo Zhang

A practical learning control system is described in this paper, which is applicable to the control of complex robotic systems. In the controller, a stochastic reinforcement learning algorithm is used to learn functions with continuous outputs as control signals. The authors present a CMAC-based network incorporating stochastic real-valued units that learns to perform an underconstrained positioning task using a simulated 2-degree-of-freedom robot arm. The authors also investigate the effects of varying learning algorithm parameters. >

ICRA Conference 1992 Conference Paper

Time-varying potential field based 'perception-action' behaviors of mobile robot

• Tianmiao Wang
• Bo Zhang

An efficient method is presented for implementing mobile robot perception-action behaviors, based on the time-varying potential field. The concept of dynamic potential is proposed to achieve the control scheme. The scheme was successfully implemented on the THMR-II mobile robot for a variety of applications. The drawbacks of known potential field methods are solved by the composition of the behavior controls. Experimental results based on sonar and vision signals are given to show that the method had good real-time response and adaptability, and was robust in dynamic environments. >

IJCAI Conference 1989 Conference Paper

Motion Planning of Multi-Joint Robotic Arm with Topological Dimension Reduction Method

• Bo Zhang
• Ling Zhang
• Tian Zhang

IJCAI Conference 1985 Conference Paper

A Weighted Technique in Heuristic Search

• Bo Zhang
• Ling Zhang

As shown in [1], we examine search as a s t a t i s t i c s a m p l i n g p r o c e s s. Based o n some s t a t i s t i c a l i n f e r e n c e method the p r o b a b i l i t y t h a t a s u b t r e e i n search t r e e c o n t a i n s the g o a l can b e d e c i d e d. Thus some w e i g h t is in t e n t i o n a l y added to t h e e v a l u a t i o n f u n c t i o n o f those nodes which are u n l i k e l y i n the s o l u t i o n p a t h s o t h a t the s e a r c h w i l l c o n c e n t r a t e o n the most p r o m i s i n g p a t h. I t r e s u l t s in a new w e i g h t e d a l g o r i t h m - W S A. Tn a u n i f o r m m-ary t r e e, we show t h a t a g o a l can be found by WSA in the p o l y n o m i a l t i m e, a l t h o u g h t h e c o m p u t a t i o n a l c o m p l e x i t y of A ( o r A*) may be 0 ( e ) f o r s e a r c h i n g the same s p a c e. Where N i s the depth a t which t h e g o a l i s l o c a t e d.

IJCAI Conference 1983 Conference Paper

The Statistical Inference Method in Heuristic Search Techniques

• Ling Zhang
• Bo Zhang