Arrow Research search

Author name cluster

Fan Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

67 papers
2 author rows

Possible papers

67

AAAI Conference 2026 Conference Paper

CyC3D: Fine-grained Controllable 3D Generation via Cycle Consistency Regularization

  • Hongbin Xu
  • Chaohui Yu
  • Feng Xiao
  • Jiazheng Xing
  • Hai Ci
  • Weitao Chen
  • Fan Wang
  • Ming Li

Despite the remarkable progress of 3D generation, achieving controllability, i.e., ensuring consistency between generated 3D content and input conditions like edge and depth, remains a significant challenge. Existing methods often struggle to maintain accurate alignment, leading to noticeable discrepancies. To address this issue, we propose CyC3D, a new framework that enhances controllable 3D generation by explicitly encouraging cyclic consistency between the second-order 3D content, generated based on extracted signals from the first-order generation, and its original input controls. Specifically, we employ an efficient feed-forward backbone that can generate a 3D object from an input condition and a text prompt. Given an initial viewpoint and a control signal, a novel view is rendered from the generated 3D content, from which the extracted condition is used to regenerate the 3D content. This re-generated output is then rendered back to the initial viewpoint, followed by another round of control signal extraction, forming a cyclic process with two consistency constraints. View consistency ensures coherence between the two generated 3D objects, measured by semantic similarity to accommodate generative diversity. Condition consistency aligns the final extracted signal with the original input control, preserving structural or geometric details throughout the process. Extensive experiments on popular benchmarks demonstrate that CyC3D significantly improves controllability, especially for fine-grained details, outperforming existing methods across various conditions (e.g., +14.17% PSNR for edge, +6.26% PSNR for sketch).

PDF Details DOI

AAAI Conference 2026 Conference Paper

EarthCrafter: Scalable 3D Earth Generation via Dual-Sparse Latent Diffusion

  • Shang Liu
  • Chenjie Cao
  • Chaohui Yu
  • Wen Qian
  • Jing Wang
  • Fan Wang

Despite the remarkable developments achieved by recent 3D generation works, scaling these methods to geographic extents, such as modeling thousands of square kilometers of Earth’s surface, remains an open challenge. We address this through a dual innovation in data infrastructure and model architecture. First, we introduce Aerial-Earth3D, the largest 3D aerial dataset to date, consisting of 50k curated scenes (each measuring 600m) captured across the U.S. mainland, comprising 45M multi-view Google Earth frames. Each scene provides pose-annotated multi-view images, depth maps, normals, semantic segmentation, and camera poses, with explicit quality control to ensure terrain diversity. Building on this foundation, we propose EarthCrafter, a tailored framework for large-scale 3D Earth generation via sparse-decoupled latent diffusion. Our architecture separates structural and textural generation: 1) Dual sparse 3D-VAEs compress high-resolution geometric voxels and textural 2D Gaussian Splats (2DGS) into compact latent spaces, largely alleviating the costly computation suffering from vast geographic scales while preserving critical information. 2) We propose condition-aware flow matching models trained on mixed inputs (semantics, images, or neither) to flexibly model latent geometry and texture features independently. Extensive experiments demonstrate that EarthCrafter performs substantially better in extremely large-scale generation. The framework further supports versatile applications, from semantic-guided urban layout generation to unconditional terrain synthesis, while maintaining geographic plausibility through our rich data priors from Aerial-Earth3D.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Retrieval-driven Reasoning for Deliberative Visual Classification

  • Jianye Xie
  • Lianyong Qi
  • Fan Wang
  • Anqi Wang
  • Wenjuan Gong
  • Danxin Wang
  • Wanchun Dou
  • Yang Cao

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in visual classification tasks. Existing methods for enhancing VLMs on this task often rely heavily on direct category-to-image matching, which limits generalization and results in suboptimal performance. In addition, these methods provide no understanding of why a specific category is chosen. To address these limitations, we introduce a new deliberative visual classification task that decomposes the classification process into multiple deliberative steps and leverages Large Language Models (LLMs) to perform explicit reasoning before the final decision. Specifically, we propose a Retrieval-driven Reasoning model (RdR) with two components, i.e., retrieval database construction and deliberative category prediction. The first component leverages LLMs to extract category-relevant descriptors and constructs a retrieval database for effective image–descriptor matching. The second component facilitates multiple deliberative steps and performs explicit reasoning based on the retrieved descriptors to augment the category prediction. Extensive experiments on multiple datasets demonstrate that RdR consistently outperforms strong baselines, highlighting its robustness and generalization ability.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Retriever Encoder Selection Matters for In-Context Learning-based Medical Segmentation

  • Fan Wang
  • Zhongyi Han
  • Yongshun Gong
  • Yilong Yin

In-context learning-based medical segmentation (ICLM) enables foundation models to generalize to unseen cases without retraining. To enhance performance on test queries, existing methods typically follow a two-stage process: (1) using a retrieval encoder (RE) to map both queries and training samples into a shared feature space, and (2) retrieving and utilizing the top-k most similar training samples. While current methods fix the RE and focus on optimizing stage (2), we show that the choice of RE in stage (1) alone can account for over 70% of the performance variation, highlighting RE selection as a critical yet often overlooked factor in ICLM. In this paper, we conduct an analysis of the RE selection and make two main findings: (1) dynamically selecting the RE for each query outperforms selecting a fixed RE for the entire task; and (2) feature-space heuristics (e.g., intra-class compactness and inter-class separability) fail to predict RE quality. To this end, we propose the instance-adaptive retrieval encoder selection (IRES) method that can select the optimal RE for each query based on output predictions. IRES is based on the intuition that a good RE retrieves relevant demonstrations, helping the ICL model generate more accurate and stable segmentation masks. Thus, we introduce the shape stability score (S³), which evaluates the morphological stability of predicted masks under iterative erosion. Experiments show S³ correlates strongly with true RE quality (Pearson > 0.8), serving as a reliable selection proxy. To reduce S³’s per-query cost, we propose parallel prediction with reciprocal neighbor reuse (P2R), which accelerates inference by parallelizing encoding and reusing encoder selections across reciprocal neighbors, avoiding redundant computation. Built on S³ and P2R, IRES improves ICLM performance across FUNDUS, Brain MRI, and Chest X-ray datasets, with up to 10.6% gain on fundus segmentation.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Subspace-Aware Graph Construction and Contrastive Alignment for Multimodal Recommendation with Large Language Models

  • Haodong Li
  • Lianyong Qi
  • Weiming Liu
  • Fan Wang
  • Chong Li
  • Shengye Pang
  • Wenwen Gong
  • Yanwei Xu

Multimedia content offers additional context for recommender systems to better understand user interests. Existing studies on multimodal recommendation primarily focus on constructing item-item semantic graphs. However, most of these methods capture only shallow semantic structures based on feature similarity and struggle to model more complex or cross-entity semantic relationships (e.g., user-item). Moreover, in these methods, collaborative signals often dominate and suppress semantic knowledge, which limits its role in representation learning. To address these issues, we propose SCALE, a novel framework that combines subspace-aware graph construction and contrastive alignment for multimodal recommendation with large language models. Specifically, we first use large language models and encoders to extract user and item features. Following the subspace clustering assumption, we apply the Orthogonal Matching Pursuit algorithm to mine complex semantic structures within the item-item, user-user, and user-item spaces, and integrate them into a unified semantic graph. We then perform graph convolution on both the semantic and interaction graphs, and aggregate the results for recommendation. Furthermore, contrastive losses are employed to enhance semantic fusion and alignment. Extensive experiments on five real-world datasets demonstrate that SCALE significantly outperforms state-of-the-art multimodal recommendation models, highlighting its effectiveness in modeling complex relationships and integrating semantic knowledge with collaborative signals.

PDF Details DOI

TAAS Journal 2025 Journal Article

Adaptive Scheduling of High-Availability Drone Swarms for Congestion Alleviation in Connected Automated Vehicles

  • Shengye Pang
  • Yi Li
  • Zhen Qin
  • Xinkui Zhao
  • Jintao Chen
  • Fan Wang
  • Jianwei Yin

The Intelligent Transportation System (ITS) serves as a pivotal element within urban networks, offering decision support to users and connected automated vehicles through comprehensive information gathering, sensing, device control, and data processing. Presently, ITS predominantly relies on sensors embedded in fixed infrastructure, notably Roadside Units (RSUs). However, RSUs are confined by coverage limitations and may encounter challenges in prompt emergency responses. On-demand resources, such as drones, present a viable option to supplement these deficiencies effectively. This article introduces an approach where Software-Defined Networking and Mobile Edge Computing technologies are integrated to formulate a high-availability drone swarm control and communication infrastructure framework comprising the cloud layer, edge layer, and device layer. Drones confront limitations in flight duration attributed to battery limitations, posing a challenge in sustaining continuous monitoring of road conditions over extended periods. Effective drone scheduling stands as a promising solution to overcome these constraints. To tackle this issue, we initially utilized Graph WaveNet, a specialized graph neural network structure tailored for spatial-temporal graph modeling, for training a congestion prediction model using real-world dataset inputs. Building upon this, we further propose an algorithm for drone scheduling based on congestion prediction. Our simulation experiments using real-world data demonstrate that, compared to the baseline method, the proposed scheduling algorithm not only yielded superior scheduling gains but also mitigated drone idle rates.

Details DOI

ICLR Conference 2025 Conference Paper

AutoCGP: Closed-Loop Concept-Guided Policies from Unlabeled Demonstrations

  • Pei Zhou
  • Ruizhe Liu
  • Qian Luo
  • Fan Wang
  • Yibing Song
  • Yanchao Yang 0001

Training embodied agents to perform complex robotic tasks presents significant challenges due to the entangled factors of task compositionality, environmental diversity, and dynamic changes. In this work, we introduce a novel imitation learning framework to train closed-loop concept-guided policies that enhance long-horizon task performance by leveraging discovered manipulation concepts. Unlike methods that rely on predefined skills and human-annotated labels, our approach allows agents to autonomously abstract manipulation concepts from their proprioceptive states, thereby alleviating misalignment due to ambiguities in human semantics and environmental complexity. Our framework comprises two primary components: an *Automatic Concept Discovery* module that identifies meaningful and consistent manipulation concepts, and a *Concept-Guided Policy Learning* module that effectively utilizes these manipulation concepts for adaptive task execution, including a *Concept Selection Transformer* for concept-based guidance and a *Concept-Guided Policy* for action prediction with the selected concepts. Experiments demonstrate that our approach significantly outperforms baseline methods across a range of tasks and environments, while showcasing emergent consistency in motion patterns associated with the discovered manipulation concepts. Codes are available at: https://github.com/PeiZhou26/AutoCGP.

Details

NeurIPS Conference 2025 Conference Paper

CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step

  • Zheyuan Liu
  • Munan Ning
  • Qihui Zhang
  • Shuo Yang
  • Zhongrui Wang
  • Yiwei Yang
  • Xianzhe Xu
  • Yibing Song

Current text-to-image (T2I) generation models struggle to align spatial composition with the input text, especially in complex scenes. Even layout-based approaches yield suboptimal spatial control, as their generation process is decoupled from layout planning, making it difficult to refine the layout during synthesis. We present CoT-Diff, a framework that brings step-by-step CoT-style reasoning into T2I generation by tightly integrating Multimodal Large Language Model (MLLM)-driven 3D layout planning with the diffusion process. CoT-Diff enables layout-aware reasoning inline within a single diffusion round: at each denoising step, the MLLM evaluates intermediate predictions, dynamically updates the 3D scene layout, and continuously guides the generation process. The updated layout is converted into semantic conditions and depth maps, which are fused into the diffusion model via a condition-aware attention mechanism, enabling precise spatial control and semantic injection. Experiments on 3D Scene benchmarks show that CoT-Diff significantly improves spatial alignment and compositional fidelity, and outperforms the state-of-the-art method by 34. 7% in complex scene spatial accuracy, thereby validating the effectiveness of this entangled generation paradigm.

PDF Details

AAAI Conference 2025 Conference Paper

DR-VAE: Debiased and Representation-enhanced Variational Autoencoder for Collaborative Recommendation

  • Fan Wang
  • Chaochao Chen
  • Weiming Liu
  • Minye Lei
  • Jintao Chen
  • Yuwen Liu
  • Xiaolin Zheng
  • Jianwei Yin

Recommender Systems (RSs) are widely applied for navigating information, and Collaborative Filtering (CF) is one of prominent recommendation techniques due to the advantages of domain independence and easy interpretation. Among the numerous CF methods, Variational Autoencoders (VAE), benefiting from modeling in a probabilitistic way, stands out in capturing user preferences through representation learning. Despite the superiority, VAE-based CF models still suffer from two challenging problems: (1) Exposure bias: models in training state are narrowly exposed to a limited, biased sample of data, leading to a skewed understanding of users' true preferences; (2) Posterior collapse: models excessively simplify the learned latent variable distributions, generating na"ive representations that are unable to encapsulate the complex data patterns and thereby resulting improper recommendations. In this paper, we propose a Debiased and Representation-enhanced Variational AutoEncoder (DR-VAE) framework for collaborative recommendations. Specifically, for exposure bias problem, DR-VAE incorporates a Debiasing Estimator, mitigating the impact of exposure bias. For poster collapse issue, DR-VAE innovatively introduces a Flow-based Representation Enhancement module, ensuring us to encapsulate complex data patterns by fitting complex and intricate posterior distributions directly. We provide experimental validations over four datasets to substantiate the efficacy of our DR-VAE framework.

PDF Details DOI

TCS Journal 2025 Journal Article

Edge-bipancyclicity of hypercubes with faulty edges

  • Qianhong Liu
  • Fan Wang

In this paper, we consider the edge-bipancyclicity property of hypercubes with faulty edges. For n ≥ 4, let Q n be an n-dimensional hypercube, and let F be a set of faulty edges in Q n such that | F | ≤ 3 n − 11. If every vertex in Q n − F has a degree of at least 2 and no 4-cycle in Q n − F has a pair of non-adjacent vertices both of degree 2, then every edge e in Q n − F lies on a fault-free cycle of every even length from 8 to 2 n, with two exceptions: (1) there exists a 4-cycle in Q n − F containing the edge e such that the vertices on the 4-cycle which are not incident with e both have a degree of 2; (2) there exist two vertices of degree 2 which are not endpoints of edge e but are adjacent to the same endpoint of e in Q n − F. This result improves upon some known results on edge-bipancyclicity of hypercubes with faulty edges.

Details DOI

JBHI Journal 2025 Journal Article

EEG Temporal-Spatial Feature Learning for Automated Selection of Stimulus Parameters in Electroconvulsive Therapy

  • Fan Wang
  • Dan Chen
  • Shenhong Weng
  • Tengfei Gao
  • Yiping Zuo
  • Yuntao Zheng

The risk of adverse effects in Electroconvulsive Therapy (ECT), such as cognitive impairment, can be high if an excessive stimulus is applied to induce the necessary generalized seizure (GS); Conversely, inadequate stimulus results in failure. Recent efforts to automate this task can facilitate statistical analyses on individual parameters or qualitative predictions. However, this automation still significantly lags behind the requirements in clinical practices. This study addresses this issue by predicting the probability of GS induction under the joint restriction of a patient's EEG (electroencephalogram) and the stimulus parameters, sustained by a two-stage learning model (namely ECTnet): 1) Temporal-Spatial Feature Learning. Channel-wise convolution via multiple convolution kernels first learns the deep features of the EEG, followed by a “ConvLSTM” constructing the temporal-spatial features aided with the enforced convolution operations at the LSTM gates; 2) GS Prediction. The probability of seizure induction is predicted based on the EEG features fused with stimulus parameters, through which the optimal parameter setting(s) may be obtained by minimizing the stimulus charge while ensuring the probability above a threshold. Experiments have been conducted on EEG data from 96 subjects with mental disorders to examine the performance and design of ECTnet. These experiments indicate that ECTnet can effectively automate the selection of optimal stimulus parameters: 1) an AUC of 0. 746, F1-score of 0. 90, a precision of 89% and a recall of 93% in the prediction of seizure induction have been achieved, outperforming the state-of-the-art counterpart, and 2) inclusion of parameter features increases the F1-score by 0. 054.

Details DOI

NeurIPS Conference 2025 Conference Paper

EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval

  • Zebin Yang
  • Sunjian Zheng
  • Tong Xie
  • Tianshi Xu
  • Bo Yu
  • Fan Wang
  • Jie Tang
  • Shaoshan Liu

Object-goal navigation (ObjNav) tasks an agent with navigating to the location of a specific object in an unseen environment. Embodied agents equipped with large language models (LLMs) and online constructed navigation maps can perform ObjNav in a zero-shot manner. However, existing agents heavily rely on giant LLMs on the cloud, e. g. , GPT-4, while directly switching to small LLMs, e. g. , LLaMA3. 2-11b, suffer from significant success rate drops due to limited model capacity for understanding complex navigation maps, which prevents deploying ObjNav on local devices. At the same time, the long prompt introduced by the navigation map description will cause high planning latency on local devices. In this paper, we propose EfficientNav to enable on-device efficient LLM-based zero-shot ObjNav. To help the smaller LLMs better understand the environment, we propose semantics-aware memory retrieval to prune redundant information in navigation maps. To reduce planning latency, we propose discrete memory caching and attention-based memory clustering to efficiently save and re-use the KV cache. Extensive experimental results demonstrate that EfficientNav achieves 11. 1\% improvement in success rate on HM3D benchmark over GPT-4-based baselines, and demonstrates 6. 7$\times$ real-time latency reduction and 4. 7$\times$ end-to-end latency reduction over GPT-4 planner. Our code is available on https: //github. com/PKU-SEC-Lab/EfficientNav.

PDF Details

AAAI Conference 2025 Conference Paper

Enhancing Diffusion Model with Auxiliary Information Mining-Exploration and Efficient Sampling Mechanism for Sequential Recommendation

  • Te Song
  • Lianyong Qi
  • Weiming Liu
  • Fan Wang
  • Xiaolong Xu
  • Xuyun Zhang
  • Amin Beheshti
  • Xiaokang Zhou

Sequential recommendation aims to capture the temporal dependencies of items in a user's historical interactions and make recommendations based on this. Previous generative methods addressed the issue of data not directly reflecting user preference uncertainty by modeling the distribution of latent item representations. Diffusion model (DM)-based methods have achieved significant success due to their high-quality generation and stable training. However, they lack satisfactory user sequence representations to guide the generation process, impacting recommendation performance. Moreover, these methods overlook the drawback of slow inference speed, severely limiting their practical value. To obtain effective generative guidance signals and accelerate the recommendation process, we propose DAE4Rec. In this approach, a Graph Auto-Encoder (GAE) is used to obtain interpretable item node representations, revealing global transitions of items that previous methods struggled to uncover. Then, we use it to construct a generative guidance signal with lower coupling and variance for the diffusion model. Additionally, by employing a non-Markov chain derived from the forward diffusion process, it is the first to implement a 'skip-step' reverse process in diffusion model-based methods. And a creatively designed compensator is used to bridge the performance gap caused by 'skip-step'. Extensive experiments on three real-world datasets demonstrate that DAE4Rec outperforms other state-of-the-art generative sequential recommenders.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?

  • Yuqian Yuan
  • Ronghao Dang
  • Long Li
  • Wentong Li
  • Dian Jiao
  • Xin Li
  • Deli Zhao
  • Fan Wang

The emergence of multimodal large language models (MLLMs) has driven breakthroughs in egocentric vision applications. These applications necessitate persistent, context-aware understanding of objects, as users interact with tools in dynamic and cluttered environments. However, existing embodied benchmarks primarily focus on static scene exploration, emphasizing object's appearance and spatial attributes while neglecting the assessment of dynamic changes arising from users' interactions. capabilities in object-level spatiotemporal reasoning required for real-world interactions. To address this gap, we introduce EOC-Bench, an innovative benchmark designed to systematically evaluate object-centric embodied cognition in dynamic egocentric scenarios. Specially, EOC-Bench features 3, 277 meticulously annotated QA pairs categorized into three temporal categories: Past, Present, and Future, covering 11 fine-grained evaluation dimensions and 3 visual object referencing types. To ensure thorough assessment, we develop a mixed-format human-in-the-loop annotation frameworkBased on EOC-Bench, we conduct comprehensive evaluations of various proprietary, open-source, and object-level MLLMs. EOC-Bench serves as a crucial tool for advancing the embodied object cognitive capabilities of MLLMs, establishing a robust foundation for developing reliable core models for embodied systems.

PDF Details

NeurIPS Conference 2025 Conference Paper

FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion

  • Akide Liu
  • Zeyu Zhang
  • Zhexin Li
  • Xuehai Bai
  • Yuanjie Xing
  • Yizeng Han
  • Jiasheng Tang
  • Jichao Wu

Diffusion generative models have become the standard for producing high-quality, coherent video content, yet their slow inference speeds and high computational demands hinder practical deployment. Although both quantization and sparsity can independently accelerate inference while maintaining generation quality, naively combining these techniques in existing training-free approaches leads to significant performance degradation, as they fail to achieve proper joint optimization. We introduce FPSAttention, a novel training-aware co-design of FP8 quantization and Sparsity for video generation, with a focus on the 3D bi-directional attention mechanism. Our approach features three key innovations: 1) A unified 3D tile-wise granularity that simultaneously supports both quantization and sparsity. 2) A denoising step-aware strategy that adapts to the noise schedule, addressing the strong correlation between quantization/sparsity errors and denoising steps. 3) A native, hardware-friendly kernel that leverages FlashAttention and is implemented with optimized Hopper architecture features, enabling highly efficient execution. Trained on Wan2. 1's 1. 3B and 14B models and evaluated on the vBench benchmark, FPSAttention achieves a 7. 09$\times$ kernel speedup for attention operations and a 4. 96$\times$ end-to-end speedup for video generation compared to the BF16 baseline at 720p resolution—without sacrificing generation quality.

PDF Details

NeurIPS Conference 2025 Conference Paper

GeoVideo: Introducing Geometric Regularization into Video Generation Model

  • Yunpeng Bai
  • Shaoheng Fang
  • Chaohui Yu
  • Fan Wang
  • Qixing Huang

Recent advances in video generation have enabled the synthesis of high-quality and visually realistic clips using diffusion transformer models. However, most existing approaches operate purely in the 2D pixel space and lack explicit mechanisms for modeling 3D structures, often resulting in temporally inconsistent geometries, implausible motions, and structural artifacts. In this work, we introduce geometric regularization losses into video generation by augmenting latent diffusion models with per-frame depth prediction. We adopted depth as the geometric representation because of the great progress in depth prediction and its compatibility with image-based latent encoders. Specifically, to enforce structural consistency over time, we propose a multi-view geometric loss that aligns the predicted depth maps across frames within a shared 3D coordinate system. Our method bridges the gap between appearance generation and 3D structure modeling, leading to improved spatio-temporal coherence, shape consistency, and physical plausibility. Experiments across multiple datasets show that our approach produces significantly more stable and geometrically consistent results than existing baselines.

PDF Details

ICRA Conference 2025 Conference Paper

MuST: Multi-Head Skill Transformer for Long-Horizon Dexterous Manipulation with Skill Progress

  • Kai Gao
  • Fan Wang
  • Erica Aduh
  • Dylan Randle
  • Jane Shi

Robot picking and packing tasks require dexterous manipulation skills, such as rearranging objects to establish a good grasping pose, or placing and pushing items to achieve tight packing. These tasks are challenging for robots due to the complexity and variability of the required actions. To tackle the difficulty of learning and executing long-horizon tasks, we propose a novel framework called the Multi-Head Skill Transformer (MuST). This model is designed to learn and sequentially chain together multiple motion primitives (skills), enabling robots to perform complex sequences of actions effectively. MuST introduces a “progress value” for each skill, guiding the robot on which skill to execute next and ensuring smooth transitions between skills. Additionally, our model is capable of expanding its skill set and managing various sequences of sub-tasks efficiently. Extensive experiments in both simulated and real-world environments demonstrate that MuST significantly enhances the robot's ability to perform long-horizon dexterous manipulation tasks.

Details

NeurIPS Conference 2025 Conference Paper

PlayerOne: Egocentric World Simulator

  • Yuanpeng Tu
  • Hao Luo
  • Xi Chen
  • Xiang Bai
  • Fan Wang
  • Hengshuang Zhao

We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real-scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and world-consistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications.

PDF Details

AAAI Conference 2025 Conference Paper

RealisHuman: A Two-Stage Approach for Refining Malformed Human Parts in Generated Images

  • Benzhi Wang
  • Jingkai Zhou
  • Jingqi Bai
  • Yang Yang
  • Weihua Chen
  • Fan Wang
  • Zhen Lei

In recent years, diffusion models have revolutionized visual generation, outperforming traditional frameworks like Generative Adversarial Networks (GANs). However, generating images of humans with realistic semantic parts, such as hands and faces, remains a significant challenge due to their intricate structural complexity. To address this issue, we propose a novel post-processing solution named RealisHuman. The RealisHuman framework operates in two stages. First, it generates realistic human parts, such as hands or faces, using the original malformed parts as references, ensuring consistent details with the original image. Second, it seamlessly integrates the rectified human parts back into their corresponding positions by repainting the surrounding areas to ensure smooth and realistic blending. The RealisHuman framework significantly enhances the realism of human generation, as demonstrated by notable improvements in both qualitative and quantitative metrics.

PDF Details DOI

AAAI Conference 2025 Conference Paper

RealisID: Scale-Robust and Fine-Controllable Identity Customization via Local and Global Complementation

  • Zhaoyang Sun
  • Fei Du
  • Weihua Chen
  • Fan Wang
  • Yaxiong Chen
  • Yi Rong
  • Shengwu Xiong

Recently, the success of text-to-image synthesis has greatly advanced the development of identity customization techniques, whose main goal is to produce realistic identity-specific photographs based on text prompts and reference face images. However, it is difficult for existing identity customization methods to simultaneously meet the various requirements of different real-world applications, including the identity fidelity of small face, the control of face location, pose and expression, as well as the customization of multiple persons. To this end, we propose a scale-robust and fine-controllable method, namely RealisID, which learns different control capabilities through the cooperation between a pair of local and global branches. Specifically, by using cropping and up-sampling operations to filter out face-irrelevant information, the local branch concentrates the fine control of facial details and the scale-robust identity fidelity within the face region. Meanwhile, the global branch manages the overall harmony of the entire image. It also controls the face location by taking the location guidance as input. As a result, RealisID can benefit from the complementarity of these two branches. Finally, by implementing our branches with two different variants of ControlNet, our method can be easily extended to handle multi-person customization, even only trained on single-person datasets. Extensive experiments and ablation studies indicate the effectiveness of RealisID and verify its ability in fulfilling all the requirements mentioned above.

PDF Details DOI

ICML Conference 2025 Conference Paper

SE(3)-Equivariant Diffusion Policy in Spherical Fourier Space

  • Xupeng Zhu
  • Fan Wang
  • Robin Walters 0001
  • Jane Shi

Diffusion Policies are effective at learning closed-loop manipulation policies from human demonstrations but generalize poorly to novel arrangements of objects in 3D space, hurting real-world performance. To address this issue, we propose Spherical Diffusion Policy (SDP), an SE(3) equivariant diffusion policy that adapts trajectories according to 3D transformations of the scene. Such equivariance is achieved by embedding the states, actions, and the denoising process in spherical Fourier space. Additionally, we employ novel spherical FiLM layers to condition the action denoising process equivariantly on the scene embeddings. Lastly, we propose a spherical denoising temporal U-net that achieves spatiotemporal equivariance with computational efficiency. In the end, SDP is end-to-end SE(3) equivariant, allowing robust generalization across transformed 3D scenes. SDP demonstrates a large performance improvement over strong baselines in 20 simulation tasks and 5 physical robot tasks including single-arm and bi-manual embodiments. Code is available at https: //github. com/amazon-science/Spherical_Diffusion_Policy.

Details

NeurIPS Conference 2025 Conference Paper

Solving Discrete (Semi) Unbalanced Optimal Transport with Equivalent Transformation Mechanism and KKT-Multiplier Regularization

  • Weiming Liu
  • Xinting Liao
  • Jun Dan
  • Fan Wang
  • Hua Yu
  • Junhao Dong
  • Shunjie Dong
  • Lianyong Qi

Semi-Unbalanced Optimal Transport (SemiUOT) shows great promise in matching two probability measures by relaxing one of the marginal constraints. Previous solvers often incorporate an entropy regularization term, which can result in inaccurate matching solutions. To address this issue, we focus on determining the marginal probability distribution of SemiUOT with KL divergence using the proposed Equivalent Transformation Mechanism (ETM) approach. Furthermore, we extend the ETM-based method into exploiting the marginal probability distribution of Unbalanced Optimal Transport (UOT) with KL divergence for validating its generalization. Once the marginal probabilities of UOT/SemiUOT are determined, they can be transformed into a classical Optimal Transport (OT) problem. Moreover, we propose a KKT-Multiplier regularization term combined with Multiplier Regularized Optimal Transport (MROT) to achieve more accurate matching results. We conduct several numerical experiments to demonstrate the effectiveness of our proposed methods in addressing UOT/SemiUOT problems.

PDF Details

NeurIPS Conference 2025 Conference Paper

SparseDiT: Token Sparsification for Efficient Diffusion Transformer

  • Shuning Chang
  • Pichao Wang
  • Jiasheng Tang
  • Fan Wang
  • Yi Yang

Diffusion Transformers (DiT) are renowned for their impressive generative performance; however, they are significantly constrained by considerable computational costs due to the quadratic complexity in self-attention and the extensive sampling steps required. While advancements have been made in expediting the sampling process, the underlying architectural inefficiencies within DiT remain underexplored. We introduce SparseDiT, a novel framework that implements token sparsification across spatial and temporal dimensions to enhance computational efficiency while preserving generative quality. Spatially, SparseDiT employs a tri-segment architecture that allocates token density based on feature requirements at each layer: Poolingformer in the bottom layers for efficient global feature extraction, Sparse-Dense Token Modules (SDTM) in the middle layers to balance global context with local detail, and dense tokens in the top layers to refine high-frequency details. Temporally, SparseDiT dynamically modulates token density across denoising stages, progressively increasing token count as finer details emerge in later timesteps. This synergy between SparseDiT’s spatially adaptive architecture and its temporal pruning strategy enables a unified framework that balances efficiency and fidelity throughout the generation process. Our experiments demonstrate SparseDiT’s effectiveness, achieving a 55\% reduction in FLOPs and a 175\% improvement in inference speed on DiT-XL with similar FID score on 512$\times$512 ImageNet, a 56\% reduction in FLOPs across video generation datasets, and a 69\% improvement in inference speed on PixArt-$\alpha$ on text-to-image generation task with a 0. 24 FID score decrease. SparseDiT provides a scalable solution for high-quality diffusion-based generation compatible with sampling optimization techniques. Code is available at https: //github. com/changsn/SparseDiT.

PDF Details

NeurIPS Conference 2025 Conference Paper

SpiderSolver: A Geometry-Aware Transformer for Solving PDEs on Complex Geometries

  • KAI QI
  • Fan Wang
  • Zhewen Dong
  • Jian Sun

Transformers have demonstrated effectiveness in solving partial differential equations (PDEs). However, extending them to solve PDEs on complex geometries remains a challenge. In this work, we propose SpiderSolver, a geometry-aware transformer that introduces spiderweb tokenization for handling complex domain geometry and irregularly discretized points. Our method partitions the irregular spatial domain into spiderweb-like patches, guided by the domain boundary geometry. SpiderSolver leverages a coarse-grained attention mechanism to capture global interactions across spiderweb tokens and a fine-grained attention mechanism to refine feature interactions between the domain boundary and its neighboring interior points. We evaluate SpiderSolver on PDEs with diverse domain geometries across seven datasets, including cars, airfoils, blood flow in the human thoracic aorta, as well as canonical cases governed by the Navier-Stokes, Darcy flow, elasticity, and plasticity equations. Experimental results demonstrate that SpiderSolver consistently achieves state-of-the-art performance across different datasets and metrics, with better generalization ability in the OOD setting. The code is available at https: //github. com/Kai-Qi/SpiderSolver.

PDF Details

NeurIPS Conference 2025 Conference Paper

Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds

  • Fan Wang
  • Pengtao Shao
  • Yiming Zhang
  • Bo Yu
  • Shaoshan Liu
  • Ning Ding
  • Yang Cao
  • Yu Kang

In-Context Reinforcement Learning (ICRL) enables agents to learn automatically and on-the-fly from their interactive experiences. However, a major challenge in scaling up ICRL is the lack of scalable task collections. To address this, we propose the procedurally generated tabular Markov Decision Processes, named AnyMDP. Through a carefully designed randomization process, AnyMDP is capable of generating high-quality tasks on a large scale while maintaining relatively low structural biases. To facilitate efficient meta-training at scale, we further introduce decoupled policy distillation and induce prior information in the ICRL framework. Our results demonstrate that, with a sufficiently large scale of AnyMDP tasks, the proposed model can generalize to tasks that were not considered in the training set through versatile in-context learning paradigms. The scalable task set provided by AnyMDP also enables a more thorough empirical investigation of the relationship between data distribution and ICRL performance. We further show that the generalization of ICRL potentially comes at the cost of increased task diversity and longer adaptation periods. This finding carries critical implications for scaling robust ICRL capabilities, highlighting the necessity of diverse and extensive task design, and prioritizing asymptotic performance over few-shot adaptation.

PDF Details

ICML Conference 2025 Conference Paper

Transfer Learning for Nonparametric Contextual Dynamic Pricing

  • Fan Wang
  • Feiyu Jiang
  • Zifeng Zhao
  • Yi Yu 0016

Dynamic pricing strategies are crucial for firms to maximize revenue by adjusting prices based on market conditions and customer characteristics. However, designing optimal pricing strategies becomes challenging when historical data are limited, as is often the case when launching new products or entering new markets. One promising approach to overcome this limitation is to leverage information from related products or markets to inform the focal pricing decisions. In this paper, we explore transfer learning for nonparametric contextual dynamic pricing under a covariate shift model, where the marginal distributions of covariates differ between source and target domains while the reward functions remain the same. We propose a novel Transfer Learning for Dynamic Pricing (TLDP) algorithm that can effectively leverage pre-collected data from a source domain to enhance pricing decisions in the target domain. The regret upper bound of TLDP is established under a simple Lipschitz condition on the reward function. To establish the optimality of TLDP, we further derive a matching minimax lower bound, which includes the target-only scenario as a special case and is presented for the first time in the literature. Extensive numerical experiments validate our approach, demonstrating its superiority over existing methods and highlighting its practical utility in real-world applications.

Details

ICRA Conference 2025 Conference Paper

Transferring Visual Knowledge: Semi-Supervised Instance Segmentation for Object Navigation Across Varying Height Viewpoints

  • Qiu Zheng
  • Junjie Hu 0003
  • Yuming Liu
  • Zengfeng Zeng
  • Fan Wang
  • Tin Lun Lam

The object navigation task requires robots to understand the semantic regularities in their environments. However, existing modular object navigation frameworks rely on instance segmentation models trained at fixed camera height viewpoints, limiting generalization performance and increasing labeling costs for new height viewpoints. To tackle this issue, we propose a semi-supervised method that transfers knowledge from a source height to a target height, minimizing the need for additional labels. Our approach introduces three key innovations: i) a projection policy to enhance the teacher model's detection capabilities at the target height, ii) a dynamic weight mechanism that emphasizes high-confidence pseudo-labels to reduce overfitting, and iii) a prototype contrast transferring method to transfer knowl-edge effectively. Experiments on the Habitat- Matterport 3D (HM3D) dataset show our method outperforms state-of-the-art semi-supervised techniques, improving both segmentation accuracy and navigation performance. The code is available at: https://github.com/FreeformRobotics/TransferKnowledge.

Details

NeurIPS Conference 2025 Conference Paper

UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback

  • Pengwei Liu
  • Hangjie Yuan
  • Bo Dong
  • Jiazheng Xing
  • Jinwang Wang
  • Rui Zhao
  • Weihua Chen
  • Fan Wang

Relighting is a crucial task with both practical demand and artistic value, and recent diffusion models have shown strong potential by enabling rich and controllable lighting effects. However, as they are typically optimized in semantic latent space, where proximity does not guarantee physical correctness in visual space, they often produce unrealistic results—such as overexposed highlights, misaligned shadows, and incorrect occlusions. We address this with UniLumos, a unified relighting framework for both images and videos that brings RGB-space geometry feedback into a flow-matching backbone. By supervising the model with depth and normal maps extracted from its outputs, we explicitly align lighting effects with the scene structure, enhancing physical plausibility. Nevertheless, this feedback requires high-quality outputs for supervision in visual space, making standard multi-step denoising computationally expensive. To mitigate this, we employ path consistency learning, allowing supervision to remain effective even under few-step training regimes. To enable fine-grained relighting control and supervision, we design a structured six-dimensional annotation protocol capturing core illumination attributes. Building upon this, we propose LumosBench, a disentangled attribute-level benchmark that evaluates lighting controllability via large vision-language models, enabling automatic and interpretable assessment of relighting precision across individual dimensions. Extensive experiments demonstrate that UniLumos achieves state-of-the-art relighting quality with significantly improved physical consistency, while delivering a 20x speedup for both image and video relighting. Code is available at https: //github. com/alibaba-damo-academy/Lumos-Custom.

PDF Details

IJCAI Conference 2025 Conference Paper

Variational Graph Auto-Encoder Driven Graph Enhancement for Sequential Recommendation

  • Yuwen Liu
  • Lianyong Qi
  • Xingyuan Mao
  • Weiming Liu
  • Shichao Pei
  • Fan Wang
  • Xuyun Zhang
  • Amin Beheshti

Recommender systems play a critical role in many applications by providing personalized recommendations based on user interactions. However, it remains a major challenge to capture complex sequential patterns and address noise in user interaction data. While advanced neural networks have enhanced sequential recommendation by modeling high-order item dependencies, they typically assume that the noisy interaction data as the user's preferred preferences. This assumption can lead to suboptimal recommendation results. We propose a Variational Graph Auto-Encoder driven Graph Enhancement (VGAE-GE) method for robust augmentation in sequential recommendation. Specifically, our method first constructs an item transition graph to capture higher-order interactions and employs a Variational Graph Auto-Encoder (VGAE) to generate latent variable distributions. By utilizing these latent variable distributions for graph reconstruction, we can improve the item representation. Next, we use a Graph Convolutional Network (GCN) to transform these latent variables into embeddings and infer more robust user representations from the updated item embeddings. Finally, we obtain the reconstructed user check-in data, and then use a Mamba-based recommender to make the recommendation process more efficient and the recommendation results more accurate. Extensive experiments on five public datasets demonstrate that our VGAE-GE model improves recommendation performance and robustness.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Animate3D: Animating Any 3D Model with Multi-view Video Diffusion

  • Yanqin Jiang
  • Chaohui Yu
  • Chenjie Cao
  • Fan Wang
  • Weiming Hu
  • Jin Gao

Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model’s multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct coarse motions directly from generated multi-view videos, followed by the introduced 4D-SDS to model fine-level motions. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models are open-released.

PDF Details DOI

IROS Conference 2024 Conference Paper

Avoiding Object Damage in Robotic Manipulation

  • Erica Aduh
  • Fan Wang
  • Dylan Randle
  • Kaiwen Wang
  • Priyesh Shah
  • Chaitanya Mitash
  • Manikantan Nambi

The large-scale deployment of robotic manipulation systems in warehouses has highlighted the rare but costly problem of robot-induced object damage. We present a system that uses a classification model to predict whether an object will get damaged during robotic manipulation. The model uses object attributes retrieved from warehouse information systems as well as attributes available at our robotic workcell. We evaluated different classical machine learning models, as well as a large language model (BERT) and a multimodal-transformer for our task. We show that the multi-modal transformer model that is able to leverage text and image data outperforms models that only rely on categorical and numerical data. Furthermore, our comparative analysis equips the selection the optimal model for an application. We validate our system during an experiment in which the output of the damage prediction system is used to avoid picking objects that are likely to get damaged. In over 50k pick-and-place activities, our system reduces damage rate by 64%.

Details

AAAI Conference 2024 Conference Paper

BVT-IMA: Binary Vision Transformer with Information-Modified Attention

  • Zhenyu Wang
  • Hao Luo
  • Xuemei Xie
  • Fan Wang
  • Guangming Shi

As a compression method that can significantly reduce the cost of calculations and memories, model binarization has been extensively studied in convolutional neural networks. However, the recently popular vision transformer models pose new challenges to such a technique, in which the binarized models suffer from serious performance drops. In this paper, an attention shifting is observed in the binary multi-head self-attention module, which can influence the information fusion between tokens and thus hurts the model performance. From the perspective of information theory, we find a correlation between attention scores and the information quantity, further indicating that a reason for such a phenomenon may be the loss of the information quantity induced by constant moduli of binarized tokens. Finally, we reveal the information quantity hidden in the attention maps of binary vision transformers and propose a simple approach to modify the attention values with look-up information tables so that improve the model performance. Extensive experiments on CIFAR-100/TinyImageNet/ImageNet-1k demonstrate the effectiveness of the proposed information-modified attention on binary vision transformers.

PDF Details DOI

AAAI Conference 2024 Conference Paper

CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer

  • Yabing Wang
  • Fan Wang
  • Jianfeng Dong
  • Hao Luo

Cross-lingual cross-modal retrieval has garnered increasing attention recently, which aims to achieve the alignment between vision and target language (V-T) without using any annotated V-T data pairs. Current methods employ machine translation (MT) to construct pseudo-parallel data pairs, which are then used to learn a multi-lingual and multi-modal embedding space that aligns visual and target-language representations. However, the large heterogeneous gap between vision and text, along with the noise present in target language translations, poses significant challenges in effectively aligning their representations. To address these challenges, we propose a general framework, Cross-Lingual to Cross-Modal (CL2CM), which improves the alignment between vision and target language using cross-lingual transfer. This approach allows us to fully leverage the merits of multi-lingual pre-trained models (e.g., mBERT) and the benefits of the same modality structure, i.e., smaller gap, to provide reliable and comprehensive semantic correspondence (knowledge) for the cross-modal network. We evaluate our proposed approach on two multilingual image-text datasets, Multi30K and MSCOCO, and one video-text dataset, VATEX. The results clearly demonstrate the effectiveness of our proposed method and its high potential for large-scale retrieval.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation

  • Wangbo Zhao
  • Jiasheng Tang
  • Yizeng Han
  • Yibing Song
  • Kai Wang
  • Gao Huang
  • Fan Wang
  • Yang You

Existing parameter-efficient fine-tuning (PEFT) methods have achieved significant success on vision transformers (ViTs) adaptation by improving parameter efficiency. However, the exploration of enhancing inference efficiency during adaptation remains underexplored. This limits the broader application of pre-trained ViT models, especially when the model is computationally extensive. In this paper, we propose Dynamic Tuning (DyT), a novel approach to improve both parameter and inference efficiency for ViT adaptation. Specifically, besides using the lightweight adapter modules, we propose a token dispatcher to distinguish informative tokens from less important ones, allowing the latter to dynamically skip the original block, thereby reducing the redundant computation during inference. Additionally, we explore multiple design variants to find the best practice of DyT. Finally, inspired by the mixture-of-experts (MoE) mechanism, we introduce an enhanced adapter to further boost the adaptation performance. We validate DyT across various tasks, including image/video recognition and semantic segmentation. For instance, DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Enhancing Dual-Target Cross-Domain Recommendation with Federated Privacy-Preserving Learning

  • Zhenghong Lin
  • Wei Huang
  • Hengyu Zhang
  • Jiayu Xu
  • Weiming Liu
  • Xinting Liao
  • Fan Wang
  • Shiping Wang

Recently, dual-target cross-domain recommendation (DTCDR) has been proposed to alleviate the data sparsity problem by sharing the common knowledge across domains simultaneously. However, existing methods often assume that personal data containing abundant identifiable information can be directly accessed, which results in a controversial privacy leakage problem of DTCDR. To this end, we introduce the P2DTR framework, a novel approach in DTCDR while protecting private user information. Specifically, we first design a novel inter-client knowledge extraction mechanism, which exploits the private set intersection algorithm and prototype-based federated learning to enable collaboratively modeling among multiple users and a server. Furthermore, to improve the recommendation performance based on the extracted common knowledge across domains, we proposed an intra-client enhanced recommendation, consisting of a constrained dominant set (CDS) propagation mechanism and dual-recommendation module. Extensive experiments on real-world datasets validate that our proposed P2DTR framework achieves superior utility under a privacy-preserving guarantee on both domains.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Exploring Channel-Aware Typical Features for Out-of-Distribution Detection

  • Rundong He
  • Yue Yuan
  • Zhongyi Han
  • Fan Wang
  • Wan Su
  • Yilong Yin
  • Tongliang Liu
  • Yongshun Gong

Detecting out-of-distribution (OOD) data is essential to ensure the reliability of machine learning models when deployed in real-world scenarios. Different from most previous test-time OOD detection methods that focus on designing OOD scores, we delve into the challenges in OOD detection from the perspective of typicality and regard the feature’s high-probability region as the feature’s typical set. However, the existing typical-feature-based OOD detection method implies an assumption: the proportion of typical feature sets for each channel is fixed. According to our experimental analysis, each channel contributes differently to OOD detection. Adopting a fixed proportion for all channels results in several channels losing too many typical features or incorporating too many abnormal features, resulting in low performance. Therefore, exploring the channel-aware typical features is crucial to better-separating ID and OOD data. Driven by this insight, we propose expLoring channel-Aware tyPical featureS (LAPS). Firstly, LAPS obtains the channel-aware typical set by calibrating the channel-level typical set with the global typical set from the mean and standard deviation. Then, LAPS rectifies the features into channel-aware typical sets to obtain channel-aware typical features. Finally, LAPS leverages the channel-aware typical features to calculate the energy score for OOD detection. Theoretical and visual analyses verify that LAPS achieves a better bias-variance trade-off. Experiments verify the effectiveness and generalization of LAPS under different architectures and OOD scores.

PDF Details DOI

EAAI Journal 2024 Journal Article

Improved YOLOv8-GD deep learning model for defect detection in electroluminescence images of solar photovoltaic modules

  • Yukang Cao
  • Dandan Pang
  • Qianchuan Zhao
  • Yi Yan
  • Yongqing Jiang
  • Chongyi Tian
  • Fan Wang
  • Julin Li

Photovoltaic defect detection is an essential aspect of research on building-distributed photovoltaic systems. Existing photovoltaic defect detection models based on deep learning, such as YOLOv5 and YOLOv8, have significantly improved the accuracy of photovoltaic defect detection. However, these models are too large, and their feature extraction ability is insufficient, leading to low detection efficiency and inability to cope with the continuous evolution of defects. Therefore, this study proposes an accurate and lightweight YOLOv8 (You Only Look Once v8) GD algorithm. The algorithm is an improved version of YOLOv8, wherein DW-Conv (DepthWise-Conv) is applied to the YOLOv8 backbone network. Moreover, convolution is replaced with the GSConv (Group-shuffle Conv) and the BiFPN (bidirectional feature pyramid network) structure is added to the architecture. Several electroluminescent photovoltaic defect datasets are used to verify the effectiveness of the proposed method. The final experimental results show that the map@0. 5 and map@0. 5 ∼ 0. 95 of YOLOv8-GD are 92. 8% and 63. 1%, respectively, which are 4. 2% and 5. 7% higher than those of the original algorithm, respectively, and the model volume is reduced by 16. 7%. Thus, the proposed algorithm shows considerable potential in the field of photovoltaic defect detection.

Details DOI

AAAI Conference 2024 Conference Paper

Learning Accurate and Bidirectional Transformation via Dynamic Embedding Transportation for Cross-Domain Recommendation

  • Weiming Liu
  • Chaochao Chen
  • Xinting Liao
  • Mengling Hu
  • Yanchao Tan
  • Fan Wang
  • Xiaolin Zheng
  • Yew Soon Ong

With the rapid development of Internet and Web techniques, Cross-Domain Recommendation (CDR) models have been widely explored for resolving the data-sparsity and cold-start problem. Meanwhile, most CDR models should utilize explicit domain-shareable information (e.g., overlapped users or items) for knowledge transfer across domains. However, this assumption may not be always satisfied since users and items are always non-overlapped in real practice. The performance of many previous works will be severely impaired when these domain-shareable information are not available. To address the aforementioned issues, we propose the Joint Preference Exploration and Dynamic Embedding Transportation model (JPEDET) in this paper which is a novel framework for solving the CDR problem when users and items are non-overlapped. JPEDET includes two main modules, i.e., joint preference exploration module and dynamic embedding transportation module. The joint preference exploration module aims to fuse rating and review information for modelling user preferences. The dynamic embedding transportation module is set to share knowledge via neural ordinary equations for dual transformation across domains. Moreover, we innovatively propose the dynamic transport flow equipped with linear interpolation guidance on barycentric Wasserstein path for achieving accurate and bidirectional transformation. Our empirical study on Amazon datasets demonstrates that JPEDET significantly outperforms the state-of-the-art models under the CDR setting.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing

  • Chenjie Cao
  • Chaohui Yu
  • Fan Wang
  • Xiangyang Xue
  • Yanwei Fu

Novel View Synthesis (NVS) and 3D generation have recently achieved prominent improvements. However, these works mainly focus on confined categories or synthetic 3D assets, which are discouraged from generalizing to challenging in-the-wild scenes and fail to be employed with 2D synthesis directly. Moreover, these methods heavily depended on camera poses, limiting their real-world applications. To overcome these issues, we propose MVInpainter, re-formulating the 3D editing as a multi-view 2D inpainting task. Specifically, MVInpainter partially inpaints multi-view images with the reference guidance rather than intractably generating an entirely novel view from scratch, which largely simplifies the difficulty of in-the-wild NVS and leverages unmasked clues instead of explicit pose conditions. To ensure cross-view consistency, MVInpainter is enhanced by video priors from motion components and appearance guidance from concatenated reference key&value attention. Furthermore, MVInpainter incorporates slot attention to aggregate high-level optical flow features from unmasked regions to control the camera movement with pose-free training and inference. Sufficient scene-level experiments on both object-centric and forward-facing datasets verify the effectiveness of MVInpainter, including diverse tasks, such as multi-view object removal, synthesis, insertion, and replacement. The project page is https: //ewrfcas. github. io/MVInpainter/.

PDF Details DOI

IROS Conference 2024 Conference Paper

Performing Efficient and Safe Deformable Package Transport Operations Using Suction Cups

  • Rishabh Shukla
  • Zeren Yu
  • Samrudh Moode
  • Omey M. Manyar
  • Fan Wang
  • Siddharth Mayya
  • Satyandra K. Gupta

Suction cups are popular for picking and transporting packages in warehouse applications. To maximize throughput, high transport speeds are desired. Many packages are deformable and may detach from the suction cups due to inertial loading if trajectories use excessive velocities. This paper introduces a novel methodology that analyzes package deformation through its curvature at the package-suction cup contact interface to generate a Factor-of-Safety (FOS) score for each waypoint in a given trajectory. By maintaining the FOS above a predetermined threshold, the trajectory planner is able to generate transport trajectories that are both safe and time-optimized. Experimental results show the method’s efficacy, demonstrating a 21. 92% reduction in transport times compared to a conservative trajectory generation. Our FOS predictor identified trajectories that ensured safe package transport with 100% accuracy across all 627 real-world experiments.

Details

NeurIPS Conference 2024 Conference Paper

SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models

  • Zhaoyang Sun
  • Shengwu Xiong
  • Yaxiong Chen
  • Fei Du
  • Weihua Chen
  • Fan Wang
  • Yi Rong

This paper studies the challenging task of makeup transfer, which aims to apply diverse makeup styles precisely and naturally to a given facial image. Due to the absence of paired data, current methods typically synthesize sub-optimal pseudo ground truths to guide the model training, resulting in low makeup fidelity. Additionally, different makeup styles generally have varying effects on the person face, but existing methods struggle to deal with this diversity. To address these issues, we propose a novel Self-supervised Hierarchical Makeup Transfer (SHMT) method via latent diffusion models. Following a "decoupling-and-reconstruction" paradigm, SHMT works in a self-supervised manner, freeing itself from the misguidance of imprecise pseudo-paired data. Furthermore, to accommodate a variety of makeup styles, hierarchical texture details are decomposed via a Laplacian pyramid and selectively introduced to the content representation. Finally, we design a novel Iterative Dual Alignment (IDA) module that dynamically adjusts the injection condition of the diffusion model, allowing the alignment errors caused by the domain gap between content and makeup representations to be corrected. Extensive quantitative and qualitative analyses demonstrate the effectiveness of our method. Our code is available at https: //github. com/Snowfallingplum/SHMT.

PDF Details DOI

IROS Conference 2024 Conference Paper

Simulation-Assisted Learning for Efficient Bin-Packing of Deformable Packages in a Bimanual Robotic Cell

  • Omey M. Manyar
  • Hantao Ye
  • Meghana Sagare
  • Siddharth Mayya
  • Fan Wang
  • Satyandra K. Gupta

Bin-packing is an important problem in the robotic warehouse domain. Traditionally, this problem has been studied only for rigid packages (e. g. , boxes or rigid objects). In this work, we tackle the problem of bin-packing with deformable packages that have become a popular choice for fulfillment needs. We present a system that incorporates a dual robot arm bimanual setup, uniquely combining suction and sweeping motions to stably and reliably pack deformable packages in a bin. Additionally, we propose a comprehensive action prediction framework to optimize for bin-packing efficiency by predicting optimal actions for both robots involved. Our methodology leverages a two-pronged learning strategy, where initially, we train a model in a self-supervised manner to predict a scoring metric indicative of bin-packing efficiency and then leverage an online optimization scheme to compute optimal actions in real time. The model is pre-trained in simulation in MuJoCo and fine-tuned on small-scale data from a real-world laboratory setting. Our packing score prediction model predicts bin-packing score ∈ [0, 1] with an MSE of 0. 003. Real-world experiments validate our method’s adaptability to novel scenarios and its effectiveness in packing operations. Project Website: https://sites.google.com/usc.edu/bimanual-binpacking/

Details

ICRA Conference 2023 Conference Paper

ARMBench: An Object-centric Benchmark Dataset for Robotic Manipulation

  • Chaitanya Mitash
  • Fan Wang
  • Shiyang Lu
  • Vikedo Terhuja
  • Tyler W. Garaas
  • Felipe Polido
  • Manikantan Nambi

This paper introduces Amazon Robotic Manipulation Benchmark (ARMBench), a large-scale, object-centric benchmark dataset for robotic manipulation in the context of a warehouse. Automation of operations in modern warehouses requires a robotic manipulator to deal with a wide variety of objects, unstructured storage, and dynamically changing inventory. Such settings pose challenges in perceiving the identity, physical characteristics, and state of objects during manipulation. Existing datasets for robotic manipulation consider a limited set of objects or utilize 3D models to generate synthetic scenes with limitation in capturing the variety of object properties, clutter, and interactions. We present a large-scale dataset collected in an Amazon warehouse using a robotic manipulator performing object singulation from containers with heterogeneous contents. ARMBench contains images, videos, and metadata that corresponds to 235K+ pick-and-place activities on 190K+ unique objects. The data is captured at different stages of manipulation, i. e. , pre-pick, during transfer, and after placement. Benchmark tasks are proposed by virtue of high-quality annotations and baseline performance evaluation are presented on three visual perception challenges, namely 1) object segmentation in clutter, 2) object identification, and 3) defect detection. ARMBench can be accessed at http://armbench.com

Details

IROS Conference 2023 Conference Paper

Can Quadruped Guide Robots be Used as Guide Dogs?

  • Luyao Wang
  • Qihe Chen
  • Yan Zhang 0122
  • Ziang Li
  • Tingmin Yan
  • Fan Wang
  • Guyue Zhou
  • Jiangtao Gong

Quadruped robots have the potential to guide blind and low vision (BLV) people due to their highly flexible locomotion and emotional value provided by their bionic forms. However, the development of quadruped guide robots rarely involves BLV users' participatory designs and evaluations. In this paper, we conducted two empirical experiments both in indoor controlled and outdoor field scenarios, exploring the benefits and drawbacks of quadruped guide robots. The results show that the nowadays commercial quadruped robots exposed significant disadvantages in usability and trust compared with wheeled robots. It is concluded that the moving gait and walking noise of quadruped robots would limit the guiding effectiveness to a certain extent, and the empathetic effect of its bionic form for BLV users could not be fully reflected. Based on the findings of wheeled robots and quadruped robots' advantages, we discuss the design implications for the future guide robot design for BLV users. This paper reports the first empirical experiment about quadruped guide robots with BLV users and preliminary explores their potential improvement space in substituting guide dogs, which can inspire the further specialized design of quadruped guide robots.

Details

NeurIPS Conference 2023 Conference Paper

Data Pruning via Moving-one-Sample-out

  • Haoru Tan
  • Sitong Wu
  • Fei Du
  • Yukang Chen
  • Zhibin Wang
  • Fan Wang
  • Xiaojuan Qi

In this paper, we propose a novel data-pruning approach called moving-one-sample-out (MoSo), which aims to identify and remove the least informative samples from the training set. The core insight behind MoSo is to determine the importance of each sample by assessing its impact on the optimal empirical risk. This is achieved by measuring the extent to which the empirical risk changes when a particular sample is excluded from the training set. Instead of using the computationally expensive leaving-one-out-retraining procedure, we propose an efficient first-order approximator that only requires gradient information from different training stages. The key idea behind our approximation is that samples with gradients that are consistently aligned with the average gradient of the training set are more informative and should receive higher scores, which could be intuitively understood as follows: if the gradient from a specific sample is consistent with the average gradient vector, it implies that optimizing the network using the sample will yield a similar effect on all remaining samples. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and achieves satisfactory performance across various settings. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios and outperforms state-of-the-art methods by a large margin across various settings.

PDF Details

AAAI Conference 2023 Conference Paper

Discriminability and Transferability Estimation: A Bayesian Source Importance Estimation Approach for Multi-Source-Free Domain Adaptation

  • Zhongyi Han
  • Zhiyan Zhang
  • Fan Wang
  • Rundong He
  • Wan Su
  • Xiaoming Xi
  • Yilong Yin

Source free domain adaptation (SFDA) transfers a single-source model to the unlabeled target domain without accessing the source data. With the intelligence development of various fields, a zoo of source models is more commonly available, arising in a new setting called multi-source-free domain adaptation (MSFDA). We find that the critical inborn challenge of MSFDA is how to estimate the importance (contribution) of each source model. In this paper, we shed new Bayesian light on the fact that the posterior probability of source importance connects to discriminability and transferability. We propose Discriminability And Transferability Estimation (DATE), a universal solution for source importance estimation. Specifically, a proxy discriminability perception module equips with habitat uncertainty and density to evaluate each sample's surrounding environment. A source-similarity transferability perception module quantifies the data distribution similarity and encourages the transferability to be reasonably distributed with a domain diversity loss. Extensive experiments show that DATE can precisely and objectively estimate the source importance and outperform prior arts by non-trivial margins. Moreover, experiments demonstrate that DATE can take the most popular SFDA networks as backbones and make them become advanced MSFDA solutions.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Frequency Domain Disentanglement for Arbitrary Neural Style Transfer

  • Dongyang Li
  • Hao Luo
  • Pichao Wang
  • Zhibin Wang
  • Shang Liu
  • Fan Wang

Arbitrary neural style transfer has been a popular research topic due to its rich application scenarios. Effective disentanglement of content and style is the critical factor for synthesizing an image with arbitrary style. The existing methods focus on disentangling feature representations of content and style in the spatial domain where the content and style components are innately entangled and difficult to be disentangled clearly. Therefore, these methods always suffer from low-quality results because of the sub-optimal disentanglement. To address such a challenge, this paper proposes the frequency mixer (FreMixer) module that disentangles and re-entangles the frequency spectrum of content and style components in the frequency domain. Since content and style components have different frequency-domain characteristics (frequency bands and frequency patterns), the FreMixer could well disentangle these two components. Based on the FreMixer module, we design a novel Frequency Domain Disentanglement (FDD) framework for arbitrary neural style transfer. Qualitative and quantitative experiments verify that the proposed method can render better stylized results compared to the state-of-the-art methods.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Head-Free Lightweight Semantic Segmentation with Linear Transformer

  • Bo Dong
  • Pichao Wang
  • Fan Wang

Existing semantic segmentation works have been mainly focused on designing effective decoders; however, the computational load introduced by the overall structure has long been ignored, which hinders their applications on resource-constrained hardwares. In this paper, we propose a head-free lightweight architecture specifically for semantic segmentation, named Adaptive Frequency Transformer (AFFormer). AFFormer adopts a parallel architecture to leverage prototype representations as specific learnable local descriptions which replaces the decoder and preserves the rich image semantics on high-resolution features. Although removing the decoder compresses most of the computation, the accuracy of the parallel structure is still limited by low computational resources. Therefore, we employ heterogeneous operators (CNN and vision Transformer) for pixel embedding and prototype representations to further save computational costs. Moreover, it is very difficult to linearize the complexity of the vision Transformer from the perspective of spatial domain. Due to the fact that semantic segmentation is very sensitive to frequency information, we construct a lightweight prototype learning block with adaptive frequency filter of complexity O(n) to replace standard self attention with O(n^2). Extensive experiments on widely adopted datasets demonstrate that AFFormer achieves superior accuracy while retaining only 3M parameters. On the ADE20K dataset, AFFormer achieves 41.8 mIoU and 4.6 GFLOPs, which is 4.4 mIoU higher than Segformer, with 45% less GFLOPs. On the Cityscapes dataset, AFFormer achieves 78.7 mIoU and 34.4 GFLOPs, which is 2.5 mIoU higher than Segformer with 72.5% less GFLOPs. Code is available at https://github.com/dongbo811/AFFormer.

PDF Details DOI

ICML Conference 2023 Conference Paper

MetaModulation: Learning Variational Feature Hierarchies for Few-Shot Learning with Fewer Tasks

  • Wenfang Sun
  • Yingjun Du
  • Xiantong Zhen
  • Fan Wang
  • Ling Wang
  • Cees G. M. Snoek

Meta-learning algorithms are able to learn a new task using previously learned knowledge, but they often require a large number of meta-training tasks which may not be readily available. To address this issue, we propose a method for few-shot learning with fewer tasks, which we call MetaModulation. The key idea is to use a neural network to increase the density of the meta-training tasks by modulating batch normalization parameters during meta-training. Additionally, we modify parameters at various neural network levels, rather than just a single layer, to increase task diversity. To account for the uncertainty caused by the reduced number of training tasks, we propose a variational MetaModulation where the modulation parameters are treated as latent variables. We also introduce learning variational feature hierarchies by the variational MetaModulation, which modulates features at all layers and can take into account task uncertainty and generate more diverse tasks. The ablation studies illustrate the advantages of utilizing a learnable task modulation at different levels and demonstrate the benefit of incorporating probabilistic variants in few-task meta-learning. Our MetaModulation and its variational variants consistently outperform state-of-the-art alternatives on four few-task meta-learning benchmarks.

Details

AAAI Conference 2023 Conference Paper

SwinRDM: Integrate SwinRNN with Diffusion Model towards High-Resolution and High-Quality Weather Forecasting

  • Lei Chen
  • Fei Du
  • Yuan Hu
  • Zhibin Wang
  • Fan Wang

Data-driven medium-range weather forecasting has attracted much attention in recent years. However, the forecasting accuracy at high resolution is unsatisfactory currently. Pursuing high-resolution and high-quality weather forecasting, we develop a data-driven model SwinRDM which integrates an improved version of SwinRNN with a diffusion model. SwinRDM performs predictions at 0.25-degree resolution and achieves superior forecasting accuracy to IFS (Integrated Forecast System), the state-of-the-art operational NWP model, on representative atmospheric variables including 500 hPa geopotential (Z500), 850 hPa temperature (T850), 2-m temperature (T2M), and total precipitation (TP), at lead times of up to 5 days. We propose to leverage a two-step strategy to achieve high-resolution predictions at 0.25-degree considering the trade-off between computation memory and forecasting accuracy. Recurrent predictions for future atmospheric fields are firstly performed at 1.40625-degree resolution, and then a diffusion-based super-resolution model is leveraged to recover the high spatial resolution and finer-scale atmospheric details. SwinRDM pushes forward the performance and potential of data-driven models for a large margin towards operational applications.

PDF Details DOI

EAAI Journal 2023 Journal Article

T-distributed stochastic neighbor embedding echo state network with state matrix dimensionality reduction for time series prediction

  • Jian Huang
  • Fan Wang
  • Liang Qiao
  • Xu Yang

Echo state network (ESN), a novel type of recurrent neural network, possesses high nonlinear mapping capability, which is particularly appropriate for time series prediction. However, the huge reservoir may lead to ill-conditioned solutions in the output weight matrix, reducing the generalization ability and prediction performance of the network. To address this issue, a t-distributed stochastic neighbor embedding ESN (TESN) is proposed in this paper to replace the initial large-scale reservoir state matrix with a low-dimensional manifold. By maintaining the local neighbor relationship of the data in the original high-dimensional space, the ill-conditioned dilemma of the output weight matrix is successfully solved. Moreover, the proposed TESN has a strong ability to preserve the global features of the data, which effectively improves the prediction performance of the network. The superiority of the TESN model is demonstrated through two benchmark prediction tasks and a practical application.

Details DOI

YNIMG Journal 2022 Journal Article

Distinct networks coupled with parietal cortex for spatial representations inside and outside the visual field

  • Bo Zhang
  • Fan Wang
  • Qi Zhang
  • Yuji Naya

Our mental representation of egocentric space is influenced by the disproportionate sensory perception of the body. Previous studies have focused on the neural architecture for egocentric representations within the visual field. However, the space representation underlying the body is still unclear. To address this problem, we applied both functional Magnitude Resonance Imaging (fMRI) and Magnetoencephalography (MEG) to a spatial-memory paradigm by using a virtual environment in which human participants remembered a target location left, right, or back relative to their own body. Both experiments showed larger involvement of the frontoparietal network in representing a retrieved target on the left/right side than on the back. Conversely, the medial temporal lobe (MTL)-parietal network was more involved in retrieving a target behind the participants. The MEG data showed an earlier activation of the MTL-parietal network than that of the frontoparietal network during retrieval of a target location. These findings suggest that the parietal cortex may represent the entire space around the self-body by coordinating two distinct brain networks.

Details DOI

TMLR Journal 2022 Journal Article

Evolving Decomposed Plasticity Rules for Information-Bottlenecked Meta-Learning

  • Fan Wang
  • Hao Tian
  • Haoyi Xiong
  • Hua Wu
  • Jie Fu
  • Yang Cao
  • Yu Kang
  • Haifeng Wang

Artificial neural networks (ANNs) are typically confined to accomplishing pre-defined tasks by learning a set of static parameters. In contrast, biological neural networks (BNNs) can adapt to various new tasks by continually updating the neural connections based on the inputs, which is aligned with the paradigm of learning effective learning rules in addition to static parameters, \textit{e.g.}, meta-learning. Among various biologically inspired learning rules, Hebbian plasticity updates the neural network weights using local signals without the guide of an explicit target function, thus enabling an agent to learn automatically without human efforts. However, typical plastic ANNs using a large amount of meta-parameters violate the nature of the genomics bottleneck and potentially deteriorate the generalization capacity. This work proposes a new learning paradigm decomposing those connection-dependent plasticity rules into neuron-dependent rules thus accommodating $\Theta(n^2)$ learnable parameters with only $\Theta(n)$ meta-parameters. We also thoroughly study the effect of different neural modulation on plasticity. Our algorithms are tested in challenging random 2D maze environments, where the agents have to use their past experiences to shape the neural connections and improve their performances for the future. The results of our experiment validate the following: 1. Plasticity can be adopted to continually update a randomly initialized RNN to surpass pre-trained, more sophisticated recurrent models, especially when coming to long-term memorization. 2. Following the genomics bottleneck, the proposed decomposed plasticity can be comparable to or even more effective than canonical plasticity rules in some instances.

PDF Details

NeurIPS Conference 2022 Conference Paper

Exploiting the Relationship Between Kendall's Rank Correlation and Cosine Similarity for Attribution Protection

  • Fan Wang
  • Adams Wai Kin Kong

Model attributions are important in deep neural networks as they aid practitioners in understanding the models, but recent studies reveal that attributions can be easily perturbed by adding imperceptible noise to the input. The non-differentiable Kendall's rank correlation is a key performance index for attribution protection. In this paper, we first show that the expected Kendall's rank correlation is positively correlated to cosine similarity and then indicate that the direction of attribution is the key to attribution robustness. Based on these findings, we explore the vector space of attribution to explain the shortcomings of attribution defense methods using $\ell_p$ norm and propose integrated gradient regularizer (IGR), which maximizes the cosine similarity between natural and perturbed attributions. Our analysis further exposes that IGR encourages neurons with the same activation states for natural samples and the corresponding perturbed samples. Our experiments on different models and datasets confirm our analysis on attribution protection and demonstrate a decent improvement in adversarial robustness.

PDF Details

IJCAI Conference 2022 Conference Paper

HCFRec: Hash Collaborative Filtering via Normalized Flow with Structural Consensus for Efficient Recommendation

  • Fan Wang
  • Weiming Liu
  • Chaochao Chen
  • Mengying Zhu
  • Xiaolin Zheng

The ever-increasing data scale of user-item interactions makes it challenging for an effective and efficient recommender system. Recently, hash-based collaborative filtering (Hash-CF) approaches employ efficient Hamming distance of learned binary representations of users and items to accelerate recommendations. However, Hash-CF often faces two challenging problems, i. e. , optimization on discrete representations and preserving semantic information in learned representations. To address the above two challenges, we propose HCFRec, a novel Hash-CF approach for effective and efficient recommendations. Specifically, HCFRec not only innovatively introduces normalized flow to learn the optimal hash code by efficiently fitting a proposed approximate mixture multivariate normal distribution, a continuous but approximately discrete distribution, but also deploys a cluster consistency preserving mechanism to preserve the semantic structure in representations for more accurate recommendations. Extensive experiments conducted on six real-world datasets demonstrate the superiority of our HCFRec compared to the state-of-art methods in terms of effectiveness and efficiency.

PDF Details DOI

AAAI Conference 2022 Conference Paper

Scaled ReLU Matters for Training Vision Transformers

  • Pichao Wang
  • Xue Wang
  • Hao Luo
  • Jingkai Zhou
  • Zhipeng Zhou
  • Fan Wang
  • Hao Li
  • Rong Jin

Vision transformers (ViTs) have been an alternative design paradigm to convolutional neural networks (CNNs). However, the training of ViTs is much harder than CNNs, as it is sensitive to the training parameters, such as learning rate, optimizer and warmup epoch. The reasons for training difficulty are empirically analysed in the paper Early Convolutions Help Transformers See Better, and the authors conjecture that the issue lies with the patchify-stem of ViT models. In this paper, we further investigate this problem and extend the above conclusion: only early convolutions do not help for stable training, but the scaled ReLU operation in the convolutional stem (conv-stem) matters. We verify, both theoretically and empirically, that scaled ReLU in conv-stem not only improves training stabilization, but also increases the diversity of patch tokens, thus boosting peak performance with a large margin via adding few parameters and flops. In addition, extensive experiments are conducted to demonstrate that previous ViTs are far from being well trained, further showing that ViTs have great potential to be a better substitute of CNNs.

PDF Details

NeurIPS Conference 2022 Conference Paper

VTC-LFC: Vision Transformer Compression with Low-Frequency Components

  • Zhenyu Wang
  • Hao Luo
  • Pichao Wang
  • Feng Ding
  • Fan Wang
  • Hao Li

Although Vision transformers (ViTs) have recently dominated many vision tasks, deploying ViT models on resource-limited devices remains a challenging problem. To address such a challenge, several methods have been proposed to compress ViTs. Most of them borrow experience in convolutional neural networks (CNNs) and mainly focus on the spatial domain. However, the compression only in the spatial domain suffers from a dramatic performance drop without fine-tuning and is not robust to noise, as the noise in the spatial domain can easily confuse the pruning criteria, leading to some parameters/channels being pruned incorrectly. Inspired by recent findings that self-attention is a low-pass filter and low-frequency signals/components are more informative to ViTs, this paper proposes compressing ViTs with low-frequency components. Two metrics named low-frequency sensitivity (LFS) and low-frequency energy (LFE) are proposed for better channel pruning and token pruning. Additionally, a bottom-up cascade pruning scheme is applied to compress different dimensions jointly. Extensive experiments demonstrate that the proposed method could save 40% ～ 60% of the FLOPs in ViTs, thus significantly increasing the throughput on practical devices with less than 1% performance drop on ImageNet-1K.

PDF Details

IS Journal 2020 Journal Article

An AI Model for Oil Volatility Forecasting

  • Fan Wang
  • Junqin Lin
  • Lijian Wei

By introducing a genetic algorithm learning with a classifier system, we construct an AI model for oil volatility forecasting on the basis of Internal Information and External Information. The model provides decision support for mark-to-market portfolio and risk management by forecasting whether 1-day-ahead volatility is above a given threshold. Moreover, we explore the dynamic influencing mechanism of different types of information through information usage frequency in the learning process. In particular, we find that the jump component of oil realized volatility is efficient only in bull market, and currency information contributes most rather than oil information in bear market. Therefore, this article provides an AI method to forecast oil volatility as well as to improve the information structure of forecasting models.

Details DOI

YNIMG Journal 2019 Journal Article

Computational neuroanatomy of baby brains: A review

  • Gang Li
  • Li Wang
  • Pew-Thian Yap
  • Fan Wang
  • Zhengwang Wu
  • Yu Meng
  • Pei Dong
  • Jaeil Kim

The first postnatal years are an exceptionally dynamic and critical period of structural, functional and connectivity development of the human brain. The increasing availability of non-invasive infant brain MR images provides unprecedented opportunities for accurate and reliable charting of dynamic early brain developmental trajectories in understanding normative and aberrant growth. However, infant brain MR images typically exhibit reduced tissue contrast (especially around 6 months of age), large within-tissue intensity variations, and regionally-heterogeneous, dynamic changes, in comparison with adult brain MR images. Consequently, the existing computational tools developed typically for adult brains are not suitable for infant brain MR image processing. To address these challenges, many infant-tailored computational methods have been proposed for computational neuroanatomy of infant brains. In this review paper, we provide a comprehensive review of the state-of-the-art computational methods for infant brain MRI processing and analysis, which have advanced our understanding of early postnatal brain development. We also summarize publically available infant-dedicated resources, including MRI datasets, computational tools, grand challenges, and brain atlases. Finally, we discuss the limitations in current research and suggest potential future research directions.

Details DOI

IJCAI Conference 2019 Conference Paper

Generating Multiple Diverse Responses with Multi-Mapping and Posterior Mapping Selection

  • Chaotao Chen
  • Jinhua Peng
  • Fan Wang
  • Jun Xu
  • Hua Wu

In human conversation an input post is open to multiple potential responses, which is typically regarded as a one-to-many problem. Promising approaches mainly incorporate multiple latent mechanisms to build the one-to-many relationship. However, without accurate selection of the latent mechanism corresponding to the target response during training, these methods suffer from a rough optimization of latent mechanisms. In this paper, we propose a multi-mapping mechanism to better capture the one-to-many relationship, where multiple mapping modules are employed as latent mechanisms to model the semantic mappings from an input post to its diverse responses. For accurate optimization of latent mechanisms, a posterior mapping selection module is designed to select the corresponding mapping module according to the target response for further optimization. We also introduce an auxiliary matching loss to facilitate the optimization of posterior mapping selection. Empirical results demonstrate the superiority of our model in generating multiple diverse and informative responses over the state-of-the-art methods.

PDF Details

TCS Journal 2019 Journal Article

Hamiltonian paths passing through prescribed edges in balanced hypercubes

  • Huazhong Lü
  • Fan Wang

The balanced hypercube was proposed as a novel interconnection network for large-scale parallel systems. It is known that the balanced hypercube is edge-bipancyclic. Given a set of edges P with | P | ≤ n − 1 and two vertices x and y in different bipartite sets, we show that there exists a Hamiltonian path from x to y passing through P in B H n if and only if P is a linear forest and, neither x nor y is an internal vertex of P, and x and y are not end-vertices of a component of P simultaneously. Consequently, the balanced hypercube contains a Hamiltonian cycle passing through a set P of at most n prescribed edges if and only if P is a linear forest. Moreover, our result improves some known results.

Details DOI

ICRA Conference 2019 Conference Paper

In-hand Object Scanning via RGB-D Video Segmentation

  • Fan Wang
  • Kris Hauser

This paper proposes a technique for 3D object scanning via in-hand manipulation, in which an object reoriented in front of a video camera with multiple grasps and regrasps. In-hand object tracking is a significant challenge under fast movement, rapid appearance changes, and occlusions. This paper proposes a novel video-segmentation-based object tracking algorithm that tracks arbitrary in-hand objects more effectively than existing techniques. It also describes a novel RGB-D in-hand object manipulation dataset consisting of several common household objects. Experiments show that the new method achieves 6% increase in accuracy compared to top performing video tracking algorithms and results in noticeably higher quality reconstructed models. Moreover, testing with a novice user on a set of 200 objects demonstrates relatively rapid construction of complete 3D object models.

Details

IJCAI Conference 2019 Conference Paper

Learning to Select Knowledge for Response Generation in Dialog Systems

  • Rongzhong Lian
  • Min Xie
  • Fan Wang
  • Jinhua Peng
  • Hua Wu

End-to-end neural models for intelligent dialogue systems suffer from the problem of generating uninformative responses. Various methods were proposed to generate more informative responses by leveraging external knowledge. However, few previous work has focused on selecting appropriate knowledge in the learning process. The inappropriate selection of knowledge could prohibit the model from learning to make full use of the knowledge. Motivated by this, we propose an end-to-end neural model which employs a novel knowledge selection mechanism where both prior and posterior distributions over knowledge are used to facilitate knowledge selection. Specifically, a posterior distribution over knowledge is inferred from both utterances and responses, and it ensures the appropriate selection of knowledge during the training process. Meanwhile, a prior distribution, which is inferred from utterances only, is used to approximate the posterior distribution so that appropriate knowledge can be selected even without responses during the inference process. Compared with the previous work, our model can better incorporate appropriate knowledge in response generation. Experiments on both automatic and human evaluation verify the superiority of our model over previous baselines.

PDF Details

ICRA Conference 2019 Conference Paper

Stable Bin Packing of Non-convex 3D Objects with a Robot Manipulator

  • Fan Wang
  • Kris Hauser

Recent progress in the field of robotic manipulation has generated interest in fully automatic object packing in warehouses. This paper proposes a formulation of the packing problem that is tailored to the automated warehousing domain. Besides minimizing waste space inside a container, the problem requires stability of the object pile during packing and the feasibility of the robot motion executing the placement plans. To address this problem, a set of constraints are formulated, and a constructive packing pipeline is proposed to solve these constraints. The pipeline is able to pack geometrically complex, non-convex objects while satisfying stability and robot packability constraints. In particular, a new 3D positioning heuristic called Heightmap-Minimization heuristic is proposed, and heightmaps are used to speed up the search. Experimental evaluation of the method is conducted with a realistic physical simulator on a dataset of scanned real-world items, demonstrating stable and high-quality packing plans compared with other 3D packing methods.

Details

TCS Journal 2018 Journal Article

A type of perfect matchings extend to hamiltonian cycles in k-ary n-cubes

  • Fan Wang
  • Wuyang Sun

Kreweras conjectured that every perfect matching in a hypercube Q n for n ≥ 2 extends to a hamiltonian cycle of Q n. Fink confirmed the conjecture to be true. The k-ary n-cube Q n k is a generalization of the hypercube. However, the analogous result does not necessarily hold for Q n k. We can find a perfect matching in Q 2 6 which is not contained in any hamiltonian cycle of Q 2 6. In this paper, we investigate the existence of a hamiltonian cycle passing through a perfect matching in Q n k. For an integer n ≥ 2 and an even integer k ≥ 6, we prove that every perfect matching in Q n k consisting of edges in the same dimension can be extended to a hamiltonian cycle of Q n k.

Details DOI

ICRA Conference 2018 Conference Paper

Robot Button Pressing in Human Environments

  • Fan Wang
  • Gerry Chen
  • Kris Hauser

In order to conduct many desirable functions, service robots will need to actuate buttons and switches that are designed for humans. This paper presents the design of a robot named SwitchIt that is small, relatively inexpensive, easily mounted on a mobile robot, and actuates buttons reliably. Its operating characteristics were developed after conducting a systematic study of buttons and switches in human environments. From this study, we develop a categorization of buttons based on a set of physical properties relevant for robots to operate them. After a human calibrates and annotates buttons in the robot's environment using a hand-held tablet, the system automatically recognizes, pushes, and detects the state of a variety of buttons. Empirical tests demonstrate that the system succeeds in operating 95. 7% of 234 total buttons/switches in an office building and a household environment.

Details

TCS Journal 2005 Journal Article

k -Center problems with minimum coverage

  • Andrew Lim
  • Brian Rodrigues
  • Fan Wang
  • Zhou Xu

In this work, we study an extension of the k-center facility location problem, where centers are required to service a minimum of clients. This problem is motivated by requirements to balance the workload of centers while allowing each center to cater to a spread of clients. We study three variants of this problem, all of which are shown to be NP -hard. In-approximation hardness and approximation algorithms with factors equal or close to the best lower bounds are provided. Generalizations, including vertex costs and vertex weights, are also studied.

Details DOI
v2026.09.13