Arrow Research search

Author name cluster

Bingbing Ni

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

38 papers
2 author rows

Possible papers

38

AAAI Conference 2025 Conference Paper

InstantSticker: Realistic Decal Blending via Disentangled Object Reconstruction

  • Yi Zhang
  • Xiaoyang Huang
  • Yishun Dou
  • Yue Shi
  • Rui Shi
  • Ye Chen
  • Bingbing Ni
  • Wenjun Zhang

We present InstantSticker, a disentangled reconstruction pipeline based on Image-Based Lighting (IBL), which focuses on highly realistic decal blending, simulates stickers attached to the reconstructed surface, and allows for instant editing and real-time rendering. To achieve stereoscopic impression of the decal, we introduce shadow factor into IBL, which can be adaptively optimized during training. This allows the shadow brightness of surfaces to be accurately decomposed rather than baked into the diffuse color, ensuring that the edited texture exhibits authentic shading. To address the issues of warping and blurriness in previous methods, we apply As-Rigid-As-Possible (ARAP) parameterization to pre-unfold a specified area of the mesh and use the local UV mapping combined with a neural texture map to enhance the ability to express high-frequency details in that area. For instant editing, we utilize the Disney BRDF model, explicitly defining material colors with 3-channel diffuse albedo. This enables instant replacement of albedo RGB values during the editing process, avoiding the prolonged optimization required in previous approaches. In our experiment, we introduce the Ratio Variance Warping (RVW) metric to evaluate the local geometric warping of the decal area. Extensive experimental results demonstrate that our method surpasses previous decal blending methods in terms of editing quality, editing speed and rendering speed, achieving the state-of-the-art.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Neural Block Compression: Variable Bitrates Feature Blocks for Texture Representation

  • Rui Shi
  • Yishun Dou
  • Zhong Zheng
  • Xiangzhong Fang
  • Wenjun Zhang
  • Bingbing Ni

The imperative for compression of material textures emerges from the critical demand for high-quality rendering, which necessitates sophisticated textures that, in turn, require substantial storage and memory resources. Thus, low-bitrate compression is crucial, especially in modern games demanding higher texture resolutions. Concurrent methodologies in texture compression predominantly employ a block-based paradigm based on color space, which inevitably leads to representational redundancies and a limited compression scope, particularly at lower bitrates. In the context of mobile devices, bandwidth during texture loading and runtime memory are major bottlenecks, making existing compression algorithms inadequate for high-resolution textures. To mitigate these limitations, we propose a novel multi-resolution texture compression scheme, Neural Block Compression (NBC), developed within the neural feature domain. Our encoding scheme is constructed on a hierarchy of multi-resolution neural feature blocks, and the key ingredient is the variable bitrates quantization scheme. It allocates higher bitrates to higher feature mip-levels and lower bitrates to lower feature mip-levels, thereby extending the concept of block compression from color domain into neural feature domain. Extensive experiments demonstrate the superior texture compression quality achieved by the proposed scheme, especially at low bitrates.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

PhysDiff-VTON: Cross-Domain Physics Modeling and Trajectory Optimization for Virtual Try-On

  • Shibin Mei
  • Bingbing Ni

We present PhysDiff-VTON, a diffusion-based framework for image-based virtual try-on that systematically addresses the dual challenges of garment deformation modeling and high-frequency detail preservation. The core innovation lies in integrating physics-inspired mechanisms into the diffusion process: a pose-guided deformable warping module simulates fabric dynamics by predicting spatial offsets conditioned on human pose semantics, while wavelet-enhanced feature decomposition explicitly preserves texture fidelity through frequency-aware attention. Further enhancing generation quality, a novel sampling strategy optimizes the denoising trajectory via least action principles, enforcing temporal coherence, spatial smoothness, and multi-scale structural consistency. Comprehensive evaluations across multiple datasets demonstrate significant improvements in both geometric plausibility and perceptual quality compared to existing approaches. The framework establishes a new paradigm for synthesizing photorealistic try-on images that adhere to physical constraints while maintaining intricate garment details, advancing the practical applicability of diffusion models in fashion technology.

PDF Details

NeurIPS Conference 2025 Conference Paper

ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On

  • Yuhan Li
  • Zhiyu Jin
  • Yifan Tong
  • Wenxiang Shang
  • Benlei Cui
  • Xuanhong Chen
  • Ran Lin
  • Bingbing Ni

Virtual footwear try-on (VFTON), a critical yet underexplored area in virtual try-on (VTON), aims to synthesize faithful try-on results given diverse footwear and model images while maintaining 3D consistency and texture authenticity. Unlike conventional garment-focused VTON methods, VFTON presents unique challenges due to (1) Data Scarcity, which arises from the difficulty of perfectly matching product shoes with models wearing the identical ones, (2) Viewpoint Misalignment, where the target foot pose and source shoe views are always misaligned, leading to incomplete texture information and detail distortion, and (3) Background-induced Color Distortion, where complex material of footwear interacts with environmental lighting, causing unintended color contamination. To address these challenges, we introduce MVShoes, a multi-view shoe try-on dataset consisting of 7305 well-annotated image triplets, covering diverse footwear categories and challenging try-on scenarios. Furthermore, we propose a dual-stream DiT architecture, ShoeFit, designed to mitigate viewpoint misalignment through Multi-View Conditioning with 3D Rotary Position Embedding, and alleviate background-induced distortion using the LayeredRefAttention which leverages background features to modulate footwear latents. The proposed framework effectively decouples shoe appearance from environmental interferences while preserving high-quality texture detail through decoupled denoising and conditioning branches. Extensive quantitative and qualitative experiments demonstrate that our method substantially improves rendering fidelity and robustness under challenging real-world product shoes, establishing a new benchmark in high-fidelity footwear try-on synthesis. The dataset and benchmark will be publicly available upon acceptance of the paper.

PDF Details

NeurIPS Conference 2024 Conference Paper

AnyFit: Controllable Virtual Try-on for Any Combination of Attire Across Any Scenario

  • Yuhan Li
  • Hao Zhou
  • Wenxiang Shang
  • Ran Lin
  • Xuanhong Chen
  • Bingbing Ni

While image-based virtual try-on has made significant strides, emerging approaches still fall short of delivering high-fidelity and robust fitting images across various scenarios, as their models suffer from issues of ill-fitted garment styles and quality degrading during the training process, not to mention the lack of support for various combinations of attire. Therefore, we first propose a lightweight, scalable, operator known as Hydra Block for attire combinations. This is achieved through a parallel attention mechanism that facilitates the feature injection of multiple garments from conditionally encoded branches into the main network. Secondly, to significantly enhance the model's robustness and expressiveness in real-world scenarios, we evolve its potential across diverse settings by synthesizing the residuals of multiple models, as well as implementing a mask region boost strategy to overcome the instability caused by information leakage in existing models. Equipped with the above design, AnyFit surpasses all baselines on high-resolution benchmarks and real-world data by a large gap, excelling in producing well-fitting garments replete with photorealistic and rich details. Furthermore, AnyFit’s impressive performance on high-fidelity virtual try-ons in any scenario from any image, paves a new path for future research within the fashion community.

PDF Details DOI

AAAI Conference 2024 Conference Paper

FocalDreamer: Text-Driven 3D Editing via Focal-Fusion Assembly

  • Yuhan Li
  • Yishun Dou
  • Yue Shi
  • Yu Lei
  • Xuanhong Chen
  • Yi Zhang
  • Peng Zhou
  • Bingbing Ni

While text-3D editing has made significant strides in leveraging score distillation sampling, emerging approaches still fall short in delivering separable, precise and consistent outcomes that are vital to content creation. In response, we introduce FocalDreamer, a framework that merges base shape with editable parts according to text prompts for fine-grained editing within desired regions. Specifically, equipped with geometry union and dual-path rendering, FocalDreamer assembles independent 3D parts into a complete object, tailored for convenient instance reuse and part-wise control. We propose geometric focal loss and style consistency regularization, which encourage focal fusion and congruent overall appearance. Furthermore, FocalDreamer generates high-fidelity geometry and PBR textures which are compatible with widely-used graphics engines. Extensive experiments have highlighted the superior editing capabilities of FocalDreamer in both quantitative and qualitative evaluations.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Intrinsic Phase-Preserving Networks for Depth Super Resolution

  • Xuanhong Chen
  • Hang Wang
  • Jialiang Chen
  • Kairui Feng
  • Jinfan Liu
  • Xiaohang Wang
  • Weimin Zhang
  • Bingbing Ni

Depth map super-resolution (DSR) plays an indispensable role in 3D vision. We discover an non-trivial spectral phenomenon: the components of high-resolution (HR) and low-resolution (LR) depth maps manifest the same intrinsic phase, and the spectral phase of RGB is a superset of them, which suggests that a phase-aware filter can assist in the precise use of RGB cues. Motivated by this, we propose an intrinsic phase-preserving DSR paradigm, named IPPNet, to fully exploit inter-modality collaboration in a mutually guided way. In a nutshell, a novel Phase-Preserving Filtering Module (PPFM) is developed to generate dynamic phase-aware filters according to the LR depth flow to filter out erroneous noisy components contained in RGB and then conduct depth enhancement via the modulation of the phase-preserved RGB signal. By stacking multiple PPFM blocks, the proposed IPPNet is capable of reaching a highly competitive restoration performance. Extensive experiments on various benchmark datasets, e.g., NYU v2, RGB-D-D, reach SOTA performance and also well demonstrate the validity of the proposed phase-preserving scheme. Code: https://github.com/neuralchen/IPPNet/.

PDF Details DOI

AAAI Conference 2023 Conference Paper

AudioEar: Single-View Ear Reconstruction for Personalized Spatial Audio

  • Xiaoyang Huang
  • Yanjun Wang
  • Yang Liu
  • Bingbing Ni
  • Wenjun Zhang
  • Jinxian Liu
  • Teng Li

Spatial audio, which focuses on immersive 3D sound rendering, is widely applied in the acoustic industry. One of the key problems of current spatial audio rendering methods is the lack of personalization based on different anatomies of individuals, which is essential to produce accurate sound source positions. In this work, we address this problem from an interdisciplinary perspective. The rendering of spatial audio is strongly correlated with the 3D shape of human bodies, particularly ears. To this end, we propose to achieve personalized spatial audio by reconstructing 3D human ears with single-view images. First, to benchmark the ear reconstruction task, we introduce AudioEar3D, a high-quality 3D ear dataset consisting of 112 point cloud ear scans with RGB images. To self-supervisedly train a reconstruction model, we further collect a 2D ear dataset composed of 2,000 images, each one with manual annotation of occlusion and 55 landmarks, named AudioEar2D. To our knowledge, both datasets have the largest scale and best quality of their kinds for public use. Further, we propose AudioEarM, a reconstruction method guided by a depth estimation network that is trained on synthetic data, with two loss functions tailored for ear data. Lastly, to fill the gap between the vision and acoustics community, we develop a pipeline to integrate the reconstructed ear mesh with an off-the-shelf 3D human body and simulate a personalized Head-Related Transfer Function (HRTF), which is the core of spatial audio rendering. Code and data are publicly available in https://github.com/seanywang0408/AudioEar.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Boosting Point Clouds Rendering via Radiance Mapping

  • Xiaoyang Huang
  • Yi Zhang
  • Bingbing Ni
  • Teng Li
  • Kai Chen
  • Wenjun Zhang

Recent years we have witnessed rapid development in NeRF-based image rendering due to its high quality. However, point clouds rendering is somehow less explored. Compared to NeRF-based rendering which suffers from dense spatial sampling, point clouds rendering is naturally less computation intensive, which enables its deployment in mobile computing device. In this work, we focus on boosting the image quality of point clouds rendering with a compact model design. We first analyze the adaption of the volume rendering formulation on point clouds. Based on the analysis, we simplify the NeRF representation to a spatial mapping function which only requires single evaluation per pixel. Further, motivated by ray marching, we rectify the the noisy raw point clouds to the estimated intersection between rays and surfaces as queried coordinates, which could avoid spatial frequency collapse and neighbor point disturbance. Composed of rasterization, spatial mapping and the refinement stages, our method achieves the state-of-the-art performance on point clouds rendering, outperforming prior works by notable margins, with a smaller model size. We obtain a PSNR of 31.74 on NeRF-Synthetic, 25.88 on ScanNet and 30.81 on DTU. Code and data are publicly available in https://github.com/seanywang0408/RadianceMapping.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Fast Fluid Simulation via Dynamic Multi-Scale Gridding

  • Jinxian Liu
  • Ye Chen
  • Bingbing Ni
  • Wei Ren
  • Zhenbo Yu
  • Xiaoyang Huang

Recent works on learning-based frameworks for Lagrangian (i.e., particle-based) fluid simulation, though bypassing iterative pressure projection via efficient convolution operators, are still time-consuming due to excessive amount of particles. To address this challenge, we propose a dynamic multi-scale gridding method to reduce the magnitude of elements that have to be processed, by observing repeated particle motion patterns within certain consistent regions. Specifically, we hierarchically generate multi-scale micelles in Euclidean space by grouping particles that share similar motion patterns/characteristics based on super-light motion and scale estimation modules. With little internal motion variation, each micelle is modeled as a single rigid body with convolution only applied to a single representative particle. In addition, a distance-based interpolation is conducted to propagate relative motion message among micelles. With our efficient design, the network produces high visual fidelity fluid simulations with the inference time to be only 4.24 ms/frame (with 6K fluid particles), hence enables real-time human-computer interaction and animation. Experimental results on multiple datasets show that our work achieves great simulation acceleration with negligible prediction error increase.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Learning Continuous Depth Representation via Geometric Spatial Aggregator

  • Xiaohang Wang
  • Xuanhong Chen
  • Bingbing Ni
  • Zhengyan Tong
  • Hang Wang

Depth map super-resolution (DSR) has been a fundamental task for 3D computer vision. While arbitrary scale DSR is a more realistic setting in this scenario, previous approaches predominantly suffer from the issue of inefficient real-numbered scale upsampling. To explicitly address this issue, we propose a novel continuous depth representation for DSR. The heart of this representation is our proposed Geometric Spatial Aggregator (GSA), which exploits a distance field modulated by arbitrarily upsampled target gridding, through which the geometric information is explicitly introduced into feature aggregation and target generation. Furthermore, bricking with GSA, we present a transformer-style backbone named GeoDSR, which possesses a principled way to construct the functional mapping between local coordinates and the high-resolution output results, empowering our model with the advantage of arbitrary shape transformation ready to help diverse zooming demand. Extensive experimental results on standard depth map benchmarks, e.g., NYU v2, have demonstrated that the proposed framework achieves significant restoration gain in arbitrary scale depth map super-resolution compared with the prior art. Our codes are available at https://github.com/nana01219/GeoDSR.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Towards Interpreting and Utilizing Symmetry Property in Adversarial Examples

  • Shibin Mei
  • Chenglong Zhao
  • Bingbing Ni
  • Shengchao Yuan

In this paper, we identify symmetry property in adversarial scenario by viewing adversarial attack in a fine-grained manner. A newly designed metric called attack proportion, is thus proposed to count the proportion of the adversarial examples misclassified between classes. We observe that the distribution of attack proportion is unbalanced as each class shows vulnerability to particular classes. Further, some class pairs correlate strongly and have the same degree of attack proportion for each other. We call this intriguing phenomenon symmetry property. We empirically prove this phenomenon is widespread and then analyze the reason behind the existence of symmetry property. This explanation, to some extent, could be utilized to understand robust models, which also inspires us to strengthen adversarial defenses.

PDF Details DOI

AAAI Conference 2022 Conference Paper

Bi-volution: A Static and Dynamic Coupled Filter

  • Xiwei Hu
  • Xuanhong Chen
  • Bingbing Ni
  • Teng Li
  • Yutian Liu

Dynamic convolution has achieved significant gain in performance and computational complexity, thanks to its powerful representation capability given limited filter number/layers. However, SOTA dynamic convolution operators are sensitive to input noises (e. g. , Gaussian noise, shot noise, e. t. c.) and lack sufficient spatial contextual information in filter generation. To alleviate this inherent weakness, we propose a lightweight and heterogeneous-structure (i. e. , static and dynamic) operator, named Bi-volution. On the one hand, Bivolution is designed as a dual-branch structure to fully leverage complementary properties of static/dynamic convolution, which endows Bi-volution more robust properties and higher performance. On the other hand, the Spatial Augmented Kernel Generation module is proposed to improve the dynamic convolution, realizing the learning of spatial context information with negligible additional computational complexity. Extensive experiments illustrate that the ResNet-50 equipped with Bi-volution achieves a highly competitive boost in performance (+2. 8% top-1 accuracy on ImageNet classification, +2. 4% box AP and +2. 2% mask AP on COCO detection and instance segmentation) while maintaining extremely low FLOPs (i. e. , ResNet50@2. 7 GFLOPs). Furthermore, our Bivolution shows better robustness than dynamic convolution against various noise and input corruptions. Our code is available at https: //github. com/neuralchen/Bivolution.

PDF Details

AAAI Conference 2022 Conference Paper

Cross-Species 3D Face Morphing via Alignment-Aware Controller

  • Xirui Yan
  • Zhenbo Yu
  • Bingbing Ni
  • Hang Wang

We address cross-species 3D face morphing (i. e. , 3D face morphing from human to animal), a novel problem with promising applications in social media and movie industry. It remains challenging how to preserve target structural information and source fine-grained facial details simultaneously. To this end, we propose an Alignment-aware 3D Face Morphing (AFM) framework, which builds semantic-adaptive correspondence between source and target faces across species, via an alignment-aware controller mesh ( Explicit Controller, EC) with explicit source/target mesh binding. Based on EC, we introduce Controller-Based Mapping (CBM), which builds semantic consistency between source and target faces according to the semantic importance of different face regions. Additionally, an inference-stage coarse-to-fine strategy is exploited to produce fine-grained meshes with rich facial details from rough meshes. Extensive experimental results in multiple people and animals demonstrate that our method produces high-quality deformation results.

PDF Details

AAAI Conference 2022 Conference Paper

Exploring Visual Context for Weakly Supervised Person Search

  • Yichao Yan
  • Jinpeng Li
  • Shengcai Liao
  • Jie Qin
  • Bingbing Ni
  • Ke Lu
  • Xiaokang Yang

Person search has recently emerged as a challenging task that jointly addresses pedestrian detection and person reidentification. Existing approaches follow a fully supervised setting where both bounding box and identity annotations are available. However, annotating identities is labor-intensive, limiting the practicability and scalability of current frameworks. This paper inventively considers weakly supervised person search with only bounding box annotations. We proposed to address this novel task by investigating three levels of context clues (i. e. , detection, memory and scene) in unconstrained natural images. The first two are employed to promote local and global discriminative capabilities, while the latter enhances clustering accuracy. Despite its simple design, our CGPS achieves 80. 0% in mAP on CUHK-SYSU, boosting the baseline model by 8. 8%. Surprisingly, it even achieves comparable performance with several supervised person search models. Our code is available at https: //github. com/ljpadam/CGPS

PDF Details

NeurIPS Conference 2022 Conference Paper

RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling

  • Xuanhong Chen
  • Kairui Feng
  • Naiyuan Liu
  • Bingbing Ni
  • Yifan Lu
  • Zhengyan Tong
  • Ziang Liu

AI-for-science approaches have been applied to solve scientific problems (e. g. , nuclear fusion, ecology, genomics, meteorology) and have achieved highly promising results. Spatial precipitation downscaling is one of the most important meteorological problem and urgently requires the participation of AI. However, the lack of a well-organized and annotated large-scale dataset hinders the training and verification of more effective and advancing deep-learning models for precipitation downscaling. To alleviate these obstacles, we present the first large-scale spatial precipitation downscaling dataset named RainNet, which contains more than 62, 400 pairs of high-quality low/high-resolution precipitation maps for over 17 years, ready to help the evolution of deep learning models in precipitation downscaling. Specifically, the precipitation maps carefully collected in RainNet cover various meteorological phenomena (e. g. , hurricane, squall), which is of great help to improve the model generalization ability. In addition, the map pairs in RainNet are organized in the form of image sequences (720 maps per month or 1 map/hour), showing complex physical properties, e. g. , temporal misalignment, temporal sparse, and fluid properties. Furthermore, two deep-learning-oriented metrics are specifically introduced to evaluate or verify the comprehensive performance of the trained model (e. g. , prediction maps reconstruction accuracy). To illustrate the applications of RainNet, 14 state-of-the-art models, including deep models and traditional approaches, are evaluated. To fully explore potential downscaling solutions, we propose an implicit physical estimation benchmark framework to learn the above characteristics. Extensive experiments demonstrate the value of RainNet in training and evaluating downscaling models. Our dataset is available at https: //neuralchen. github. io/RainNet/.

PDF Details

ICLR Conference 2022 Conference Paper

Representation-Agnostic Shape Fields

  • Xiaoyang Huang
  • Jiancheng Yang
  • Yanjun Wang
  • Ziyu Chen
  • Linguo Li
  • Teng Li 0001
  • Bingbing Ni
  • Wenjun Zhang 0001

3D shape analysis has been widely explored in the era of deep learning. Numerous models have been developed for various 3D data representation formats, e.g., MeshCNN for meshes, PointNet for point clouds and VoxNet for voxels. In this study, we present Representation-Agnostic Shape Fields (RASF), a generalizable and computation-efficient shape embedding module for 3D deep learning. RASF is implemented with a learnable 3D grid with multiple channels to store local geometry. Based on RASF, shape embeddings for various 3D shape representations (point clouds, meshes and voxels) are retrieved by coordinate indexing. While there are multiple ways to optimize the learnable parameters of RASF, we provide two effective schemes among all in this paper for RASF pre-training: shape reconstruction and normal estimation. Once trained, RASF becomes a plug-and-play performance booster with negligible cost. Extensive experiments on diverse 3D representation formats, networks and applications, validate the universal effectiveness of the proposed RASF. Code and pre-trained models are publicly available\footnote{\url{https://github.com/seanywang0408/RASF}}.

Details

AAAI Conference 2022 Conference Paper

Towards Bridging Sample Complexity and Model Capacity

  • Shibin Mei
  • Chenglong Zhao
  • Shengchao Yuan
  • Bingbing Ni

In this paper, we give a new definition for sample complexity, and further develop a theoretical analysis to bridge the gap between sample complexity and model capacity. In contrast to previous works which study on some toy samples, we conduct our analysis on more general data space, and build a qualitative relationship from sample complexity to model capacity required to achieve comparable performance. Besides, we introduce a simple indicator to evaluate the sample complexity based on continuous mapping. Moreover, we further analysis the relationship between sample complexity and data distribution, which paves the way to understand the present representation learning. Extensive experiments on several datasets well demonstrate the effectiveness of our evaluation method.

PDF Details

IJCAI Conference 2021 Conference Paper

Context-Aware Image Inpainting with Learned Semantic Priors

  • Wendong Zhang
  • Junwei Zhu
  • Ying Tai
  • Yunbo Wang
  • Wenqing Chu
  • Bingbing Ni
  • Chengjie Wang
  • Xiaokang Yang

Recent advances in image inpainting have shown impressive results for generating plausible visual details on rather simple backgrounds. However, for complex scenes, it is still challenging to restore reasonable contents as the contextual information within the missing regions tends to be ambiguous. To tackle this problem, we introduce pretext tasks that are semantically meaningful to estimating the missing contents. In particular, we perform knowledge distillation on pretext models and adapt the features to image inpainting. The learned semantic priors ought to be partially invariant between the high-level pretext task and low-level image inpainting, which not only help to understand the global context but also provide structural guidance for the restoration of local textures. Based on the semantic priors, we further propose a context-aware image inpainting model, which adaptively integrates global semantics and local features in a unified image generator. The semantic learner and the image generator are trained in an end-to-end manner. We name the model SPL to highlight its ability to learn and leverage semantic priors. It achieves the state of the art on Places2, CelebA, and Paris StreetView datasets

PDF Details DOI

NeurIPS Conference 2021 Conference Paper

Joint Modeling of Visual Objects and Relations for Scene Graph Generation

  • Minghao Xu
  • Meng Qu
  • Bingbing Ni
  • Jian Tang

An in-depth scene understanding usually requires recognizing all the objects and their relations in an image, encoded as a scene graph. Most existing approaches for scene graph generation first independently recognize each object and then predict their relations independently. Though these approaches are very efficient, they ignore the dependency between different objects as well as between their relations. In this paper, we propose a principled approach to jointly predict the entire scene graph by fully capturing the dependency between different objects and between their relations. Specifically, we establish a unified conditional random field (CRF) to model the joint distribution of all the objects and their relations in a scene graph. We carefully design the potential functions to enable relational reasoning among different objects according to knowledge graph embedding methods. We further propose an efficient and effective algorithm for inference based on mean-field variational inference, in which we first provide a warm initialization by independently predicting the objects and their relations according to the current model, followed by a few iterations of relational reasoning. Experimental results on both the relationship retrieval and zero-shot relationship retrieval tasks prove the efficiency and efficacy of our proposed approach.

PDF Details

JBHI Journal 2021 Journal Article

Reinventing 2D Convolutions for 3D Images

  • Jiancheng Yang
  • Xiaoyang Huang
  • Yi He
  • Jingwei Xu
  • Canqian Yang
  • Guozheng Xu
  • Bingbing Ni

There have been considerable debates over 2D and 3D representation learning on 3D medical images. 2D approaches could benefit from large-scale 2D pretraining, whereas they are generally weak in capturing large 3D contexts. 3D approaches are natively strong in 3D contexts, however few publicly available 3D medical dataset is large and diverse enough for universal 3D pretraining. Even for hybrid (2D + 3D) approaches, the intrinsic disadvantages within the 2D/3D parts still exist. In this study, we bridge the gap between 2D and 3D convolutions by reinventing the 2D convolutions. We propose ACS (axial-coronal-sagittal) convolutions to perform natively 3D representation learning, while utilizing the pretrained weights on 2D datasets. In ACS convolutions, 2D convolution kernels are split by channel into three parts, and convoluted separately on the three views (axial, coronal and sagittal) of 3D representations. Theoretically, ANY 2D CNN (ResNet, DenseNet, or DeepLab) is able to be converted into a 3D ACS CNN, with pretrained weight of a same parameter size. Extensive experiments validate the consistent superiority of the pretrained ACS CNNs, over the 2D/3D CNN counterparts with/without pretraining. Even without pretraining, the ACS convolution can be used as a plug-and-play replacement of standard 3D convolution, with smaller model size and less computation.

Details DOI

ICML Conference 2021 Conference Paper

Self-supervised Graph-level Representation Learning with Local and Global Structure

  • Minghao Xu
  • Hang Wang
  • Bingbing Ni
  • Hongyu Guo
  • Jian Tang 0005

This paper studies unsupervised/self-supervised whole-graph representation learning, which is critical in many tasks such as molecule properties prediction in drug and material discovery. Existing methods mainly focus on preserving the local similarity structure between different graph instances but fail to discover the global semantic structure of the entire data set. In this paper, we propose a unified framework called Local-instance and Global-semantic Learning (GraphLoG) for self-supervised whole-graph representation learning. Specifically, besides preserving the local similarities, GraphLoG introduces the hierarchical prototypes to capture the global semantic clusters. An efficient online expectation-maximization (EM) algorithm is further developed for learning the model. We evaluate GraphLoG by pre-training it on massive unlabeled graphs followed by fine-tuning on downstream tasks. Extensive experiments on both chemical and biological benchmark data sets demonstrate the effectiveness of the proposed approach.

Details

AAAI Conference 2021 Conference Paper

Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale

  • Zhengyan Tong
  • Xuanhong Chen
  • Bingbing Ni
  • Xiaohang Wang

We propose a novel image-to-pencil translation method that could not only generate high-quality pencil sketches but also offer the drawing process. Existing pencil sketch algorithms are based on texture rendering rather than the direct imitation of strokes, making them unable to show the drawing process but only a final result. To address this challenge, we first establish a pencil stroke imitation mechanism. Next, we develop a framework with three branches to guide stroke drawing: the first branch guides the direction of the strokes, the second branch determines the shade of the strokes, and the third branch enhances the details further. Under this framework’s guidance, we can produce a pencil sketch by drawing one stroke every time. Our method is fully interpretable. Comparison with existing pencil drawing algorithms shows that our method is superior to others in terms of texture quality, style, and user evaluation. Our code and supplementary material are now available at: https: //github. com/TZYSJTU/Sketch-Generation-with- Drawing-Process-Guided-by-Vector-Flow-and-Grayscale

PDF Details

AAAI Conference 2020 Conference Paper

Adversarial Domain Adaptation with Domain Mixup

  • Minghao Xu
  • Jian Zhang
  • Bingbing Ni
  • Teng Li
  • Chengjie Wang
  • Qi Tian
  • Wenjun Zhang

Recent works on domain adaptation reveal the effectiveness of adversarial learning on filling the discrepancy between source and target domains. However, two common limitations exist in current adversarial-learning-based methods. First, samples from two domains alone are not sufficient to ensure domain-invariance at most part of latent space. Second, the domain discriminator involved in these methods can only judge real or fake with the guidance of hard label, while it is more reasonable to use soft scores to evaluate the generated images or features, i. e. , to fully utilize the inter-domain information. In this paper, we present adversarial domain adaptation with domain mixup (DM-ADA), which guarantees domain-invariance in a more continuous latent space and guides the domain discriminator in judging samples’ difference relative to source and target domains. Domain mixup is jointly conducted on pixel and feature level to improve the robustness of models. Extensive experiments prove that the proposed approach can achieve superior performance on tasks with various degrees of domain shift and data complexity.

PDF Details

NeurIPS Conference 2020 Conference Paper

Learning Black-Box Attackers with Transferable Priors and Query Feedback

  • Jiancheng Yang
  • Yangzhou Jiang
  • Xiaoyang Huang
  • Bingbing Ni
  • Chenglong Zhao

This paper addresses the challenging black-box adversarial attack problem, where only classification confidence of a victim model is available. Inspired by consistency of visual saliency between different vision models, a surrogate model is expected to improve the attack performance via transferability. By combining transferability-based and query-based black-box attack, we propose a surprisingly simple baseline approach (named SimBA++) using the surrogate model, which significantly outperforms several state-of-the-art methods. Moreover, to efficiently utilize the query feedback, we update the surrogate model in a novel learning scheme, named High-Order Gradient Approximation (HOGA). By constructing a high-order gradient computation graph, we update the surrogate model to approximate the victim model in both forward and backward pass. The SimBA++ and HOGA result in Learnable Black-Box Attack (LeBA), which surpasses previous state of the art by considerable margins: the proposed LeBA significantly reduces queries, while keeping higher attack success rates close to 100% in extensive ImageNet experiments, including attacking vision benchmarks and defensive models. Code is open source at https: //github. com/TrustworthyDL/LeBA.

PDF Details

ICML Conference 2020 Conference Paper

Video Prediction via Example Guidance

  • Jingwei Xu 0005
  • Huazhe Xu
  • Bingbing Ni
  • Xiaokang Yang 0001
  • Trevor Darrell

In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that can efficiently predict plausible future states, where the key insight is that the potential distribution of a sequence could be approximated with analogous ones in a repertoire of training pool, namely, expert examples. By further incorporating a novel optimization scheme into the training procedure, plausible predictions can be sampled efficiently from distribution constructed from the retrieved examples. Meanwhile, our method could be seamlessly integrated with existing stochastic predictive models; significant enhancement is observed with comprehensive experiments in both quantitative and qualitative aspects. We also demonstrate the generalization ability to predict the motion of unseen class, i. e. , without access to corresponding data during training phase. Project Page: \hyperlink{https: //sites. google. com/view/vpeg-supp/home. }{https: //sites. google. com/view/vpeg-supp/home. }

Details

AAAI Conference 2019 Conference Paper

Efficient Quantization for Neural Networks with Binary Weights and Low Bitwidth Activations

  • Kun Huang
  • Bingbing Ni
  • Xiaokang Yang

Quantization has shown stunning efficiency on deep neural network, especially for portable devices with limited resources. Most existing works uncritically extend weight quantization methods to activations. However, we take the view that best performance can be obtained by applying different quantization methods to weights and activations respectively. In this paper, we design a new activation function dubbed CReLU from the quantization perspective and further complement this design with appropriate initialization method and training procedure. Moreover, we develop a specific quantization strategy in which we formulate the forward and backward approximation of weights with binary values and quantize the activations to low bitwdth using linear or logarithmic quantizer. We show, for the first time, our final quantized model with binary weights and ultra low bitwidth activations outperforms the previous best models by large margins on ImageNet as well as achieving nearly a 10. 85× theoretical speedup with ResNet-18. Furthermore, ablation experiments and theoretical analysis demonstrate the effectiveness and robustness of CReLU in comparison with other activation functions.

PDF Details

AAAI Conference 2019 Conference Paper

Human Action Transfer Based on 3D Model Reconstruction

  • Shanyan Guan
  • Shuo Wen
  • Dexin Yang
  • Bingbing Ni
  • Wendong Zhang
  • Jun Tang
  • Xiaokang Yang

We present a practical and effective method for human action transfer. Given a sequence of source action and limited target information, we aim to transfer motion from source to target. Although recent works based on GAN or VAE achieved impressive results for action transfer in 2D, there still exists a lot of problems which cannot be avoided, such as distorted and discontinuous human body shape, blurry cloth texture and so on. In this paper, we try to solve these problems in a novel 3D viewpoint. On the one hand, we design a skeleton-to-3D-mesh generator to generate the 3D model, which achieves huge improvement on appearance reconstruction. Furthermore, we add a temporal connection to improve the smoothness of the model. On the other hand, instead of directly utilizing the image in RGB space, we transform the target appearance information into UV space for further pose transformation. Specially, unlike conventional graphics render method directly projects visible pixels to UV space, our transformation is according to pixel’s semantic information. We perform experiments on Human3. 6M and HumanEva-I to evaluate the performance of pose generator. Both qualitative and quantitative results show that our method outperforms methods based on generation method in 2D. Additionally, we compare our render method with graphic methods on Human3. 6M and People-snapshot. The comparison results show that our render method is more robust and effective.

PDF Details

AAAI Conference 2018 Conference Paper

Action Recognition With Coarse-to-Fine Deep Feature Integration and Asynchronous Fusion

  • Weiyao Lin
  • Chongyang Zhang
  • Ke Lu
  • Bin Sheng
  • Jianxin Wu
  • Bingbing Ni
  • Xin Liu
  • Hongkai Xiong

Action recognition is an important yet challenging task in computer vision. In this paper, we propose a novel deepbased framework for action recognition, which improves the recognition accuracy by: 1) deriving more precise features for representing actions, and 2) reducing the asynchrony between different information streams. We first introduce a coarse-to-fine network which extracts shared deep features at different action class granularities and progressively integrates them to obtain a more accurate feature representation for input actions. We further introduce an asynchronous fusion network. It fuses information from different streams by asynchronously integrating stream-wise features at different time points, hence better leveraging the complementary information in different streams. Experimental results on action recognition benchmarks demonstrate that our approach achieves the state-of-the-art performance.

PDF Details

IJCAI Conference 2018 Conference Paper

Human Motion Generation via Cross-Space Constrained Sampling

  • Zhongyue Huang
  • Jingwei Xu
  • Bingbing Ni

We aim to automatically generate human motion sequence from a single input person image, with some specific action label. To this end, we propose a cross-space human motion video generation network which features two paths: a forward path that first samples/generates a sequence of low dimensional motion vectors based on Gaussian Process (GP), which is paired with the input person image to form a moving human figure sequence; and a backward path based on the predicted human images to re-extract the corresponding latent motion representations. As lack of supervision, the reconstructed latent motion representations are expected to be as close as possible to the GP sampled ones, thus yielding a cyclic objective function for cross-space (i. e. , motion and appearance) mutual constrained generation. We further propose an alternative sampling/generation algorithm with respect to constraints from both spaces. Extensive experimental results show that the proposed framework successfully generates novel human motion sequences with reasonable visual quality.

PDF Details

IJCAI Conference 2018 Conference Paper

Live Face Verification with Multiple Instantialized Local Homographic Parameterization

  • Chen Lin
  • Zhouyingcheng Liao
  • Peng Zhou
  • Jianguo Hu
  • Bingbing Ni

State-of-the-art live face verification methods would easily be attacked by recorded facial expression sequence. This work directly addresses this issue via proposing a patch-wise motion parameterization based verification network infrastructure. This method directly explores the underlying subtle motion difference between the facial movements re-captured from a planer screen (e. g. , a pad) and those from a real face; therefore interactive facial expression is no longer required. Furthermore, inspired by the fact that? a fake facial movement sequence MUST contains many patch-wise fake sequences? , we embed our network into a multiple instance learning framework, which further enhance the recall rate of the proposed technique. Extensive experimental results on several face benchmarks well demonstrate the superior performance of our method.

PDF Details

NeurIPS Conference 2018 Conference Paper

Video Prediction via Selective Sampling

  • Jingwei Xu
  • Bingbing Ni
  • Xiaokang Yang

Most adversarial learning based video prediction methods suffer from image blur, since the commonly used adversarial and regression loss pair work rather in a competitive way than collaboration, yielding compromised blur effect. In the meantime, as often relying on a single-pass architecture, the predictor is inadequate to explicitly capture the forthcoming uncertainty. Our work involves two key insights: (1) Video prediction can be approached as a stochastic process: we sample a collection of proposals conforming to possible frame distribution at following time stamp, and one can select the final prediction from it. (2) De-coupling combined loss functions into dedicatedly designed sub-networks encourages them to work in a collaborative way. Combining above two insights we propose a two-stage network called VPSS (\textbf{V}ideo \textbf{P}rediction via \textbf{S}elective \textbf{S}ampling). Specifically a \emph{Sampling} module produces a collection of high quality proposals, facilitated by a multiple choice adversarial learning scheme, yielding diverse frame proposal set. Subsequently a \emph{Selection} module selects high possibility candidates from proposals and combines them to produce final prediction. Extensive experiments on diverse challenging datasets demonstrate the effectiveness of proposed video prediction approach, i. e. , yielding more diverse proposals and accurate prediction results.

PDF Details

AAAI Conference 2018 Conference Paper

Video Summarization via Semantic Attended Networks

  • Huawei Wei
  • Bingbing Ni
  • Yichao Yan
  • Huanyu Yu
  • Xiaokang Yang
  • Chen Yao

The goal of video summarization is to distill a raw video into a more compact form without losing much semantic information. However, previous methods mainly consider the diversity and representation interestingness of the obtained summary, and they seldom pay sufficient attention to semantic information of resulting frame set, especially the long temporal range semantics. To explicitly address this issue, we propose a novel technique which is able to extract the most semantically relevant video segments (i. e. , valid for a long term temporal duration) and assemble them into an informative summary. To this end, we develop a semantic attended video summarization network (SASUM) which consists of a frame selector and video descriptor to select an appropriate number of video shots by minimizing the distance between the generated description sentence of the summarized video and the human annotated text of the original video. Extensive experiments show that our method achieves a superior performance gain over previous methods on two benchmark datasets.

PDF Details

IJCAI Conference 2017 Conference Paper

Image Matching via Loopy RNN

  • Donghao Luo
  • Bingbing Ni
  • Yichao Yan
  • Xiaokang Yang

Most existing matching algorithms are one-off algorithms, i. e. , they usually measure the distance between the two image feature representation vectors for only one time. In contrast, human's vision system achieves this task, i. e. , image matching, by recursively looking at specific/related parts of both images and then making the final judgement. Towards this end, we propose a novel loopy recurrent neural network (Loopy RNN), which is capable of aggregating relationship information of two input images in a progressive/iterative manner and outputting the consolidated matching score in the final iteration. A Loopy RNN features two uniqueness. First, built on conventional long short-term memory (LSTM) nodes, it links the output gate of the tail node to the input gate of the head node, thus it brings up symmetry property required for matching. Second, a monotonous loss designed for the proposed network guarantees increasing confidence during the recursive matching process. Extensive experiments on several image matching benchmarks demonstrate the great potential of the proposed method.

PDF Details

IJCAI Conference 2017 Conference Paper

Predicting Human Interaction via Relative Attention Model

  • Yichao Yan
  • Bingbing Ni
  • Xiaokang Yang

Predicting human interaction is challenging as the on-going activity has to be inferred based on a partially observed video. Essentially, a good algorithm should effectively model the mutual influence between the two interacting subjects. Also, only a small region in the scene is discriminative for identifying the on-going interaction. In this work, we propose a relative attention model to explicitly address these difficulties. Built on a tri-coupled deep recurrent structure representing both interacting subjects and global interaction status, the proposed network collects spatio-temporal information from each subject, rectified with global interaction information, yielding effective interaction representation. Moreover, the proposed network also unifies an attention module to assign higher importance to the regions which are relevant to the on-going action. Extensive experiments have been conducted on two public datasets, and the results demonstrate that the proposed relative attention network successfully predicts informative regions between interacting subjects, which in turn yields superior human interaction prediction accuracy.

PDF Details

AAAI Conference 2017 Conference Paper

Unsupervised Deep Learning for Optical Flow Estimation

  • Zhe Ren
  • Junchi Yan
  • Bingbing Ni
  • Bin Liu
  • Xiaokang Yang
  • Hongyuan Zha

Recent work has shown that optical flow estimation can be formulated as a supervised learning problem. Moreover, convolutional networks have been successfully applied to this task. However, supervised flow learning is obfuscated by the shortage of labeled training data. As a consequence, existing methods have to turn to large synthetic datasets for easily computer generated ground truth. In this work, we explore if a deep network for flow estimation can be trained without supervision. Using image warping by the estimated flow, we devise a simple yet effective unsupervised method for learning optical flow, by directly minimizing photometric consistency. We demonstrate that a flow network can be trained from endto-end using our unsupervised scheme. In some cases, our results come tantalizingly close to the performance of methods trained with full supervision.

PDF Details

TIST Journal 2016 Journal Article

Multitask Low-Rank Affinity Graph for Image Segmentation and Image Annotation

  • Teng Li
  • Bin Cheng
  • Bingbing Ni
  • Guangchan Liu
  • Shuicheng Yan

This article investigates a low-rank representation--based graph, which can used in graph-based vision tasks including image segmentation and image annotation. It naturally fuses multiple types of image features in a framework named multitask low-rank affinity pursuit. Given the image patches described with multiple types of features, we aim at inferring a unified affinity matrix that implicitly encodes the relations among these patches. This is achieved by seeking the sparsity-consistent low-rank affinities from the joint decompositions of multiple feature matrices into pairs of sparse and low-rank matrices, the latter of which is expressed as the production of the image feature matrix and its corresponding image affinity matrix. The inference process is formulated as a minimization problem and solved efficiently with the augmented Lagrange multiplier method. Considering image patches as vertices, a graph can be built based on the resulted affinity matrix. Compared to previous methods, which are usually based on a single type of feature, the proposed method seamlessly integrates multiple types of features to jointly produce the affinity matrix in a single inference step. The proposed method is applied to graph-based image segmentation and graph-based image annotation. Experiments on benchmark datasets well validate the superiority of using multiple features over single feature and also the superiority of our method over conventional methods for feature fusion.

Details DOI

TIST Journal 2011 Journal Article

Recognizing pair-activities by causality analysis

  • Yue Zhou
  • Bingbing Ni
  • Shuicheng Yan
  • Thomas S. Huang

In this article, beyond solo-activity analysis for single object, we study the more complicated pair-activity recognition problem by exploring the relationship between two active objects based on their trajectory clues obtained from video sensor. Our contributions are three-fold. First, we design two sets of features for representing the pair-activities encoded as length-variable trajectory pairs. One set characterizes the strength of causality between two trajectories, for example, the causality ratio and feedback ratio based on the Granger Causality Test (GCT), and another set describes the style of causality between two trajectories, for example, the sampled frequency responses of the digital filter with these two trajectories as the input and output discrete signals respectively. These features along with conventional velocity and position features of a trajectory-pair are essentially of multi-modalities, and may be greatly different in scales and importance. To make full use of them, we then develop a novel feature fusing procedure to learn the coefficients for weighting these features by maximizing the discriminating power measured by weighted correlation. Finally, we collected a pair-activity database of five popular categories, each of which consists of about 170 instances. The extensive experiments on this database validate the effectiveness of the designed features for pair-activity representation, and also demonstrate that the proposed feature fusing procedure significantly boosts the pair-activity classification accuracy.

Details DOI
v2026.09.13