Jingjing Li

AAAI Conference 2026 Conference Paper

Generalizing Vision-Language Models with Dedicated Prompt Guidance

• Xinyao Li
• Yinjie Min
• Hongbo Chen
• Zhekai Du
• Fengling Li
• Jingjing Li

Fine-tuning large pretrained vision-language models (VLMs) has emerged as a prevalent paradigm for downstream adaptation, yet it faces a critical trade-off between domain specificity and domain generalization (DG) ability. Current methods typically fine-tune a universal model on the entire dataset, which potentially compromises the ability to generalize to unseen domains. To fill this gap, we provide a theoretical understanding of the generalization ability for VLM fine-tuning, which reveals that training multiple parameter-efficient expert models on partitioned source domains leads to better generalization than fine-tuning a universal model. Inspired by this finding, we propose a two-step domain-expert-Guided DG (GuiDG) framework. GuiDG first employs prompt tuning to obtain source domain experts, then introduces a Cross-Modal Attention module to guide the fine-tuning of the vision encoder via adaptive expert integration. To better evaluate few-shot DG, we construct ImageNet-DG from ImageNet and its variants. Extensive experiments on standard DG benchmarks and ImageNet-DG demonstrate that GuiDG improves upon state-of-the-art fine-tuning methods while maintaining efficiency.

AAAI Conference 2026 Conference Paper

Stable and Adaptive Fusion for Multi-domain Multi-task Recommendation

• Ke Fei
• Da Luo
• Kangyi Lin
• Zibin Zhang
• Jingjing Li

Multi-Domain Multi-Task (MDMT) recommendation aims to provide personalized recommendations by leveraging information across multiple domains and tasks. However, existing methods often suffer from spurious correlations between irrelevant features and the target, leading to negative transfer. To address this, we propose a Stable and Adaptive Fusion (SAF) framework for MDMT recommendation. SAF introduces a weighted Hilbert-Schmidt Independence Criterion (HSIC) loss to decorrelate irrelevant features from the target, learning sample weights that promote stable (i.e., robust to spurious correlations) representations in both bottom and expert layers. We employ Random Fourier Features (RFF) to enable scalable computation of the HSIC loss. We further employ adaptive feature and expert gating to select these stable features, enabling the model to capture intricate cross-domain and cross-task dependencies. The learned sample weights are also used to reweight the MDMT loss during training. Experiments on large-scale datasets show that SAF outperforms state-of-the-art baselines by up to 2% in AUC. To facilitate further research, we release a new industrial dataset with 30 million interactions across 3 domains and 2 tasks, with 300 features.

AAAI Conference 2025 Conference Paper

Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval

• Xin Huang
• Shilong Wang
• Tong Jia
• Zhihang Gou
• Jingjing Li

In the era of big data, cross-modal retrieval is increasingly important in research and application. Given the latent complexity and non-intuitive nature of cross-modal relationships, leveraging external knowledge such as large models has become a popular approach to facilitate modality alignment. Existing methods typically address these challenges by fine-tuning model encoders or using a fixed number of prompts. However, these approaches struggle with the significant information asymmetry between image-text pairs and the high distribution diversity of image data. These limitations not only introduce noise during training but also reduce the accuracy and generalization capabilities in cross-modal retrieval tasks. To address the above issues, this paper proposes Adaptive Prompt-Based Semantic Embedding with Inspired Potential of Implicit Knowledge (APSE-IPIK). On one hand, we propose an inspired potential strategy to extract fine-grained and multi-perspective text descriptions from large-scale pre-trained multimodal models, which can be seen as implicit knowledge injection. These descriptions are integrated into the visual-semantic embedding through cross-modal semantic alignment with images, balancing the information asymmetry between modalities and reducing the embedding of inaccurate mapping relationships. On the other hand, we construct an instance-level query-based prompt pool strategy to adaptively extract the most relevant prompts, addressing alignment biases caused by intra-modal (especially image) data diversity and improving alignment accuracy. Extensive experiments are conducted on two widely used datasets, Flickr30k and MSCOCO, which show the effectiveness of the proposed method.

EAAI Journal 2025 Journal Article

Design of an intelligent grading system for Chinese water chestnuts utilizing advanced artificial intelligence methods

• Yinping Zhang
• Joon Huang Chuah
• Anis Salwa Mohd Khairuddin
• Dongyang Chen
• Xuewei Zhao
• Junwei Huang
• Chenyang Xia
• Jingjing Li

This study aims to overcome limitations in Chinese water chestnut production and processing by developing an innovative grading system that integrated advanced artificial intelligence and engineering design. The artificial intelligence component features an improved convolutional neural network architecture, specifically an enhanced version of the MobileNetV2 architecture combined with a Squeeze-and-Excitation module, which is referred to as MobileSeNet. Compared to the baseline MobileNetV2 model, MobileSeNet achieved superior performance not only in grading accuracy (97. 25 %) but also in convergence stability and model generalization, benefiting from enhanced gradient flow and feature recalibration. The engineering component involved a purpose-built hardware platform that evaluated Chinese water chestnuts based on external visual traits using advanced image acquisition, preprocessing, segmentation, edge detection, and ellipse fitting for diameter estimation. A user-friendly host computer interface was incorporated to support real-time monitoring and user interaction. The system introduced in this study offered a low-cost and scalable solution, making it well-suited for implementation in real agricultural scenarios. Overall, this research advances global smart agriculture by delivering a robust and efficient grading solution underpinned by artificial intelligence, with strong adaptability to diverse agricultural products.

AAAI Conference 2025 Conference Paper

Dynamic Target Distribution Estimation for Source-Free Open-Set Domain Adaptation

• Zhiqi Yu
• Zhichao Liao
• Jingjing Li
• Zhi Chen
• Lei Zhu

Unsupervised domain adaptation (UDA) has emerged as a promising technique for transferring knowledge from a labeled domain to an unlabeled domain. However, existing UDA methods are severely constrained by data privacy and semantic inconsistencies. To alleviate these limitations, this work challenges the Source-Free Open-Set Domain Adaptation (SF-OSDA), where the pre-trained source model is directly leveraged on the open target domain for adaptation. For this purpose, we introduce the novel Dynamic Target Distribution Estimation (DTDE) method, which effectively performs known classification and unknown separation through self-supervised learning with prototypes. To construct known prototypes, a self-adaptive sampling strategy is employed to consider the category disparity. For unknown prototypes, we utilize a self-splitting and excluding principle to bypass the unknown semantics problem. Specifically, self-splitting is to evaluate the overall clustering distribution of the target domain. By excluding clusters resembling known prototypes, the remaining cluster centroids can serve as unknown prototypes. The superiority of our approach is validated across multiple benchmarks. Remarkably, DTDE outperforms the best competitor by 7.6% on the VisDA dataset.

AAAI Conference 2025 Conference Paper

Entire-Space Variational Information Exploitation for Post-Click Conversion Rate Prediction

• Ke Fei
• Xinyue Zhang
• Jingjing Li

In recommender systems, post-click conversion rate (CVR) estimation is an essential task to model user preferences for items and estimate the value of recommendations. Sample selection bias (SSB) and data sparsity (DS) are two persistent challenges for post-click conversion rate (CVR) estimation. Currently, entire-space approaches that exploit unclicked samples through knowledge distillation are promising to mitigate SSB and DS simultaneously. Existing methods use non-conversion, conversion, or adaptive conversion predictors to generate pseudo labels for unclicked samples. However, they fail to consider the unbiasedness and information limitations of these pseudo labels. Motivated by such analysis, we propose an entire-space variational information exploitation framework (EVI) for CVR prediction. First, EVI uses a conditional entire-space CVR teacher to generate unbiased pseudo labels. Then, it applies variational information exploitation and logit distillation to transfer non-click space information to the target CVR estimator. We conduct extensive offline experiments on six large-scale datasets. EVI demonstrated a 2.25% average improvement compared to the state-of-the-art baselines.

JBHI Journal 2025 Journal Article

Guest Editorial: Domain Adaptation and Generalization for Biomedical and Health Informatics

• Zheng Zhang
• Jingjing Li
• Xiaojun Chang
• Yudong Zhang

ICML Conference 2025 Conference Paper

SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior

• Jingjing Li
• Valentina Pyatkin
• Max Kleiman-Weiner
• Liwei Jiang
• Nouha Dziri
• Anne Collins
• Jana Schaich Borg
• Maarten Sap

The ideal AI safety moderation system would be both structurally interpretable (so its decisions can be reliably explained) and steerable (to align to safety standards and reflect a community’s values), which current systems fall short on. To address this gap, we present SafetyAnalyst, a novel AI safety moderation framework. Given an AI behavior, SafetyAnalyst uses chain-of-thought reasoning to analyze its potential consequences by creating a structured "harm-benefit tree, " which enumerates harmful and beneficial actions and effects the AI behavior may lead to, along with likelihood, severity, and immediacy labels that describe potential impacts on stakeholders. SafetyAnalyst then aggregates all effects into a harmfulness score using 28 fully interpretable weight parameters, which can be aligned to particular safety preferences. We applied this framework to develop an open-source LLM prompt safety classification system, distilled from 18. 5 million harm-benefit features generated by frontier LLMs on 19k prompts. On comprehensive benchmarks, we show that SafetyAnalyst (average F1=0. 81) outperforms existing moderation systems (average F1$<$0. 72) on prompt safety classification, while offering the additional advantages of interpretability, transparency, and steerability.

ICML Conference 2025 Conference Paper

SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and O(T) Complexity

• Shihao Zou
• Qingfeng Li
• Wei Ji
• Jingjing Li
• Yongkui Yang
• Guoqi Li
• Chao Dong

Spiking Neural Networks (SNNs) have shown competitive performance to Artificial Neural Networks (ANNs) in various vision tasks, while offering superior energy efficiency. However, existing SNN-based Transformers primarily focus on single-image tasks, emphasizing spatial features while not effectively leveraging SNNs’ efficiency in video-based vision tasks. In this paper, we introduce SpikeVideoFormer, an efficient spike-driven video Transformer, featuring linear temporal complexity $\mathcal{O}(T)$. Specifically, we design a spike-driven Hamming attention (SDHA) which provides a theoretically guided adaptation from traditional real-valued attention to spike-driven attention. Building on SDHA, we further analyze various spike-driven space-time attention designs and identify an optimal scheme that delivers appealing performance for video tasks, while maintaining only linear temporal complexity. The generalization ability and efficiency of our model are demonstrated across diverse downstream video tasks, including classification, human pose tracking, and semantic segmentation. Empirical results show our method achieves state-of-the-art (SOTA) performance compared to existing SNN approaches, with over 15% improvement on the latter two tasks. Additionally, it matches the performance of recent ANN-based methods while offering significant efficiency gains, achieving $\times 16$, $\times 10$ and $\times 5$ improvements on the three tasks. https: //github. com/JimmyZou/SpikeVideoFormer

ICRA Conference 2024 Conference Paper

Active Collision-Based Navigation for Wheeled Robots

• Jingjing Li
• Jialin Ji
• Qianhao Wang
• Huan Yu 0002
• Yu Pan
• Fei Gao 0011

Collision is typically avoided in robot navigation for safety guarantee. However, when a robot’s exteroceptive sensors fail, which means it becomes "blind", collision can actually be leveraged to improve localization performance. Our research demonstrates the informative nature of collisions in this context. Moreover, we show that a robot is able to navigate in a known environment with only proprioceptive sensors by actively colliding with its surroundings for more reliable localization. Firstly, we design a collision-based observation model, which is differentiable and can be easily applied to various estimators. Secondly, we integrate this model into a collision-aided localization framework and implement it in two widely used estimators, the Kalman filter and the particle filter. Thirdly, we propose an active collision path planning method, which effectively reduces localization uncertainty.

AAAI Conference 2024 Conference Paper

Agile Multi-Source-Free Domain Adaptation

• Xinyao Li
• Jingjing Li
• Fengling Li
• Lei Zhu
• Ke Lu

Efficiently utilizing rich knowledge in pretrained models has become a critical topic in the era of large models. This work focuses on adaptively utilize knowledge from multiple source-pretrained models to an unlabeled target domain without accessing the source data. Despite being a practically useful setting, existing methods require extensive parameter tuning over each source model, which is computationally expensive when facing abundant source domains or larger source models. To address this challenge, we propose a novel approach which is free of the parameter tuning over source backbones. Our technical contribution lies in the Bi-level ATtention ENsemble (Bi-ATEN) module, which learns both intra-domain weights and inter-domain ensemble weights to achieve a fine balance between instance specificity and domain consistency. By slightly tuning source bottlenecks, we achieve comparable or even superior performance on a challenging benchmark DomainNet with less than 3% trained parameters and 8 times of throughput compared with SOTA method. Furthermore, with minor modifications, the proposed module can be easily equipped to existing methods and gain more than 4% performance boost. Code is available at https://github.com/TL-UESTC/Bi-ATEN.

AAAI Conference 2024 Conference Paper

Effective Comparative Prototype Hashing for Unsupervised Domain Adaptation

• Hui Cui
• Lihai Zhao
• Fengling Li
• Lei Zhu
• Xiaohui Han
• Jingjing Li

Unsupervised domain adaptive hashing is a highly promising research direction within the field of retrieval. It aims to transfer valuable insights from the source domain to the target domain while maintaining high storage and retrieval efficiency. Despite its potential, this field remains relatively unexplored. Previous methods usually lead to unsatisfactory retrieval performance, as they frequently directly apply slightly modified domain adaptation algorithms to hash learning framework, or pursue domain alignment within the Hamming space characterized by limited semantic information. In this paper, we propose a simple yet effective approach named Comparative Prototype Hashing (CPH) for unsupervised domain adaptive image retrieval. We establish a domain-shared unit hypersphere space through prototype contrastive learning and then obtain the Hamming hypersphere space via mapping from the shared hypersphere. This strategy achieves a cohesive synergy between learning uniformly distributed and category conflict-averse feature representations, eliminating domain discrepancies, and facilitating hash code learning. Moreover, by leveraging dual-domain information to supervise the entire hashing model training process, we can generate hash codes that retain inter-sample similarity relationships within both domains. Experimental results validate that our CPH significantly outperforms the state-of-the-art counterparts across multiple cross-domain and single-domain retrieval tasks. Notably, on Office-Home and Office-31 datasets, CPH achieves an average performance improvement of 19.29% and 13.85% on cross-domain retrieval tasks compared to the second-best results, respectively. The source codes of our method are available at: https://github.com/christinecui/CPH.

NeurIPS Conference 2024 Conference Paper

Unleashing Multispectral Video's Potential in Semantic Segmentation: A Semi-supervised Viewpoint and New UAV-View Benchmark

• Wei Ji
• Jingjing Li
• Wenbo Li
• Yilin Shen
• Li Cheng
• Hongxia Jin

Thanks to the rapid progress in RGB & thermal imaging, also known as multispectral imaging, the task of multispectral video semantic segmentation, or MVSS in short, has recently drawn significant attentions. Noticeably, it offers new opportunities in improving segmentation performance under unfavorable visual conditions such as poor light or overexposure. Unfortunately, there are currently very few datasets available, including for example MVSeg dataset that focuses purely toward eye-level view; and it features the sparse annotation nature due to the intensive demands of labeling process. To address these key challenges of the MVSS task, this paper presents two major contributions: the introduction of MVUAV, a new MVSS benchmark dataset, and the development of a dedicated semi-supervised MVSS baseline - SemiMV. Our MVUAV dataset is captured via Unmanned Aerial Vehicles (UAV), which offers a unique oblique bird’s-eye view complementary to the existing MVSS datasets; it also encompasses a broad range of day/night lighting conditions and over 30 semantic categories. In the meantime, to better leverage the sparse annotations and extra unlabeled RGB-Thermal videos, a semi-supervised learning baseline, SemiMV, is proposed to enforce consistency regularization through a dedicated Cross-collaborative Consistency Learning (C3L) module and a denoised temporal aggregation strategy. Comprehensive empirical evaluations on both MVSeg and MVUAV benchmark datasets have showcased the efficacy of our SemiMV baseline.

AAAI Conference 2023 Conference Paper

Cross-Domain Adaptative Learning for Online Advertisement Customer Lifetime Value Prediction

• Hongzu Su
• Zhekai Du
• Jingjing Li
• Lei Zhu
• Ke Lu

Accurate estimation of customer lifetime value (LTV), which reflects the potential consumption of a user over a period of time, is crucial for the revenue management of online advertising platforms. However, predicting LTV in real-world applications is not an easy task since the user consumption data is usually insufficient within a specific domain. To tackle this problem, we propose a novel cross-domain adaptative framework (CDAF) to leverage consumption data from different domains. The proposed method is able to simultaneously mitigate the data scarce problem and the distribution gap problem caused by data from different domains. To be specific, our method firstly learns a LTV prediction model from a different but related platform with sufficient data provision. Subsequently, we exploit domain-invariant information to mitigate data scarce problem by minimizing the Wasserstein discrepancy between the encoded user representations of two domains. In addition, we design a dual-predictor schema which not only enhances domain-invariant information in the semantic space but also preserves domain-specific information for accurate target prediction. The proposed framework is evaluated on five datasets collected from real historical data on the advertising platform of Tencent Games. Experimental results verify that the proposed framework is able to significantly improve the LTV prediction performance on this platform. For instance, our method can boost DCNv2 with the improvement of 13.7% in terms of AUC on dataset G2. Code: https://github.com/TL-UESTC/CDAF.

NeurIPS Conference 2023 Conference Paper

Diffusion-Based Probabilistic Uncertainty Estimation for Active Domain Adaptation

• Zhekai Du
• Jingjing Li

Active Domain Adaptation (ADA) has emerged as an attractive technique for assisting domain adaptation by actively annotating a small subset of target samples. Most ADA methods focus on measuring the target representativeness beyond traditional active learning criteria to handle the domain shift problem, while leaving the uncertainty estimation to be performed by an uncalibrated deterministic model. In this work, we introduce a probabilistic framework that captures both data-level and prediction-level uncertainties beyond a point estimate. Specifically, we use variational inference to approximate the joint posterior distribution of latent representation and model prediction. The variational objective of labeled data can be formulated by a variational autoencoder and a latent diffusion classifier, and the objective of unlabeled data can be implemented in a knowledge distillation framework. We utilize adversarial learning to ensure an invariant latent space. The resulting diffusion classifier enables efficient sampling of all possible predictions for each individual to recover the predictive distribution. We then leverage a t-test-based criterion upon the sampling and select informative unlabeled target samples based on the p-value, which encodes both prediction variability and cross-category ambiguity. Experiments on both ADA and Source-Free ADA settings show that our method provides more calibrated predictions than previous ADA methods and achieves favorable performance on three domain adaptation datasets.

NeurIPS Conference 2023 Conference Paper

DVSOD: RGB-D Video Salient Object Detection

• Jingjing Li
• Wei Ji
• Size Wang
• Wenbo Li
• Li Cheng

Salient object detection (SOD) aims to identify standout elements in a scene, with recent advancements primarily focused on integrating depth data (RGB-D) or temporal data from videos to enhance SOD in complex scenes. However, the unison of two types of crucial information remains largely underexplored due to data constraints. To bridge this gap, we in this work introduce the DViSal dataset, fueling further research in the emerging field of RGB-D video salient object detection (DVSOD). Our dataset features 237 diverse RGB-D videos alongside comprehensive annotations, including object and instance-level markings, as well as bounding boxes and scribbles. These resources enable a broad scope for potential research directions. We also conduct benchmarking experiments using various SOD models, affirming the efficacy of multimodal video input for salient object detection. Lastly, we highlight some intriguing findings and promising future research avenues. To foster growth in this field, our dataset and benchmark results are publicly accessible at: https: //dvsod. github. io/.

AAAI Conference 2023 Conference Paper

Graph Component Contrastive Learning for Concept Relatedness Estimation

• Yueen Ma
• Zixing Song
• Xuming Hu
• Jingjing Li
• Yifei Zhang
• Irwin King

Concept relatedness estimation (CRE) aims to determine whether two given concepts are related. Existing methods only consider the pairwise relationship between concepts, while overlooking the higher-order relationship that could be encoded in a concept-level graph structure. We discover that this underlying graph satisfies a set of intrinsic properties of CRE, including reflexivity, commutativity, and transitivity. In this paper, we formalize the CRE properties and introduce a graph structure named ConcreteGraph. To address the data scarcity issue in CRE, we introduce a novel data augmentation approach to sample new concept pairs from the graph. As it is intractable for data augmentation to fully capture the structural information of the ConcreteGraph due to a large amount of potential concept pairs, we further introduce a novel Graph Component Contrastive Learning framework to implicitly learn the complete structure of the ConcreteGraph. Empirical results on three datasets show significant improvement over the state-of-the-art model. Detailed ablation studies demonstrate that our proposed approach can effectively capture the high-order relationship among concepts.

IJCAI Conference 2022 Conference Paper

A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model

• Xin Sun
• Tao Ge
• Shuming Ma
• Jingjing Li
• Furu Wei
• Houfeng Wang

Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited error-corrected patterns. In this paper, we propose a generic and language-independent strategy for multilingual GEC, which can train a GEC system effectively for a new non-English language with only two easy-to-access resources: 1) a pre-trained cross-lingual language model (PXLM) and 2) parallel translation data between English and the language. Our approach creates diverse parallel GEC data without any language-specific operations by taking the non-autoregressive translation generated by PXLM and the gold translation as error-corrected sentence pairs. Then, we reuse PXLM to initialize the GEC model and pre-train it with the synthetic data generated by itself, which yields further improvement. We evaluate our approach on three public benchmarks of GEC in different languages. It achieves the state-of-the-art results on the NLPCC 2018 Task 2 dataset (Chinese) and obtains competitive performance on Falko-Merlin (German) and RULEC-GEC (Russian). Further analysis demonstrates that our data construction method is complementary to rule-based approaches.

EAAI Journal 2022 Journal Article

Cross-domain Mutual Information Adversarial Maximization

• Lichao Meng
• Hongzu Su
• Chunwei Lou
• Jingjing Li

Domain adaptation challenges the problem where the source domain and the target domain have distinctive data distributions. Different from previous approaches which align the two domains by minimizing a distribution metric, in this paper, we report a new perspective of handling unsupervised domain adaptation. Specifically, we formulate domain adaptation as maximizing the obtained knowledge of the target domain through observing the source domain. Technically, we maximize the mutual information between the source domain features and the target domain features in a deep adversarial network. Firstly, we use a feature extraction network and a domain discriminator with opposite goals to form adversarial components, and learn the domain-invariant features between the source and target domains through adversarial training. Secondly, we use the optimization goal of maximizing the mutual information between cross-domain features to supervise the adversarial training process to ensure that the maximum target domain information can be obtained by observing the source domain features. Finally, we evaluate our method on four datasets: Office-31, ImageCLEF-DA, Office-Home, and VisDA-2017, and all achieve better performance than previous methods. We show that our method, named Cross-domain Mutual Information Adversarial Maximization (CMIAM), is a promising approach and able to outperform previous state-of-the-arts on various unsupervised domain adaptation tasks.

IJCAI Conference 2022 Conference Paper

Online ECG Emotion Recognition for Unknown Subjects via Hypergraph-Based Transfer Learning

• Yalan Ye
• Tongjie Pan
• Qianhe Meng
• Jingjing Li
• Li Lu

Electrocardiogram (ECG) signal based cross-subject emotion recognition methods reduce the influence of individual differences using domain adaptation (DA) techniques. These methods generally assume that the entire unlabeled data of unknown target subjects are available in training phase. However, this assumption does not hold in some practical scenarios where the data of target subjects arrive one by one in an online manner instead of being acquired at a time. Thus, existing DA methods cannot be directly applied in this case since the unknown target data is inaccessible in training phase. To tackle the problem, we propose a novel online cross-subject ECG emotion recognition method leveraging hypergraph-based online transfer learning (HOTL). Specifically, the proposed hypergraph structure is capable of learning the high-order correlation among data, such that the recognition model trained on source subjects can be more effectively generalized to target subjects. Meanwhile, the structure can be easily updated by adding a hyperedge which connects a newly coming sample with the current hypergraph, resulting in further reduce the individual differences in online manner without re-training the model. Consequently, HOTL can effectively deal with the online cross-subject scenario where unknown target ECG data arrive one by one and varying overtime. Extensive experiments conducted on the Amigos dataset validate the superiority of the proposed method.

ICLR Conference 2022 Conference Paper

Promoting Saliency From Depth: Deep Unsupervised RGB-D Saliency Detection

• Wei Ji 0011
• Jingjing Li
• Qi Bi
• Chuan Guo 0002
• Jie Liu 0044
• Li Cheng 0001

Growing interests in RGB-D salient object detection (RGB-D SOD) have been witnessed in recent years, owing partly to the popularity of depth sensors and the rapid progress of deep learning techniques. Unfortunately, existing RGB-D SOD methods typically demand large quantity of training images being thoroughly annotated at pixel-level. The laborious and time-consuming manual annotation has become a real bottleneck in various practical scenarios. On the other hand, current unsupervised RGB-D SOD methods still heavily rely on handcrafted feature representations. This inspires us to propose in this paper a deep unsupervised RGB-D saliency detection approach, which requires no manual pixel-level annotation during training. It is realized by two key ingredients in our training pipeline. First, a depth-disentangled saliency update (DSU) framework is designed to automatically produce pseudo-labels with iterative follow-up refinements, which provides more trustworthy supervision signals for training the saliency network. Second, an attentive training strategy is introduced to tackle the issue of noisy pseudo-labels, by properly re-weighting to highlight the more reliable pseudo-labels. Extensive experiments demonstrate the superior efficiency and effectiveness of our approach in tackling the challenging unsupervised RGB-D SOD scenarios. Moreover, our approach can also be adapted to work in fully-supervised situation. Empirical studies show the incorporation of our approach gives rise to notably performance improvement in existing supervised RGB-D SOD models.

YNICL Journal 2022 Journal Article

Silent brain infarction is associated with carotid siphon calcification in ischemic stroke patients

• Jingjing Li
• Yuhua Fan
• Jian Zhang
• Shihui Xing
• Shujin Tang
• Xiaoshuang Li
• Chao Dang
• Jinsheng Zeng

BACKGROUND: Silent brain infarction (SBI) had a higher prevalence in ischemic stroke patients than healthy population. Intracranial artery calcification, as the important component of atherosclerosis, is a known risk factor of ischemic stroke. Whether it is also the risk factor of SBI is uncertain. We aimed to assess the association between SBI and carotid siphon calcification (CSC) in ischemic stroke patients. METHODS: We retrospectively collected consecutive data of acute ischemic stroke patients with and without SBI by Magnetic Resonance Imaging (MRI) and calcification using non-contrast Computerized Tomography (NCCT). We used a histopathologically validated method to score the circularity, thickness, and morphology of calcification. Clinical characteristics, prevalence and pattern (intimal and medial) of CSC were compared between patients with and without SBI. The association of CSC and SBI was investigated by logistic regression analysis. RESULTS: Totally, 303 acute ischemic stroke patients were enrolled, of whom 260 (85.8%) had CSC. Patients with SBI were older (64.5 ± 10.4 years vs. 61.3 ± 12.1 years, P = 0.032), had a higher proportion of hypertension (77.5% vs. 65.7%, P = 0.035). Of the 260 CSC patients, there's no significant difference except for hyperlipidemia between patients with SBI and without SBI. The prevalence of intimal pattern of CSC was higher in those with SBI (adjusted odds ratio 2.42, 95% CI 1.219-4.794). CONCLUSIONS: Patients with SBI at acute phase of ischemic stroke have more risk factors than mentioned previously. SBI associated with the intimal pattern of CSC which relate to the atherosclerosis process in symptomatic ischemic stroke patients.

AAAI Conference 2022 Conference Paper

Text Revision By On-the-Fly Representation Optimization

• Jingjing Li
• Zichao Li
• Tao Ge
• Irwin King
• Michael R. Lyu

Text revision refers to a family of natural language generation tasks, where the source and target sequences share moderate resemblance in surface form but differentiate in attributes, such as text formality and simplicity. Current state-of-theart methods formulate these tasks as sequence-to-sequence learning problems, which rely on large-scale parallel training corpus. In this paper, we present an iterative in-place editing approach for text revision, which requires no parallel data. In this approach, we simply fine-tune a pre-trained Transformer with masked language modeling and attribute classification. During inference, the editing at each iteration is realized by two-step span replacement. At the first step, the distributed representation of the text optimizes on the fly towards an attribute function. At the second step, a text span is masked and another new one is proposed conditioned on the optimized representation. The empirical experiments on two typical and important text revision tasks, text formalization and text simplification, show the effectiveness of our approach. It achieves competitive and even better performance than state-of-the-art supervised methods on text simplification, and gains better performance than strong unsupervised methods on text formalization. Our code and model are released at https: //github. com/jingjingli01/OREO.

NeurIPS Conference 2022 Conference Paper

Variational Model Perturbation for Source-Free Domain Adaptation

• Mengmeng Jing
• Xiantong Zhen
• Jingjing Li
• Cees Snoek

We aim for source-free domain adaptation, where the task is to deploy a model pre-trained on source domains to target domains. The challenges stem from the distribution shift from the source to the target domain, coupled with the unavailability of any source data and labeled target data for optimization. Rather than fine-tuning the model by updating the parameters, we propose to perturb the source model to achieve adaptation to target domains. We introduce perturbations into the model parameters by variational Bayesian inference in a probabilistic framework. By doing so, we can effectively adapt the model to the target domain while largely preserving the discriminative ability. Importantly, we demonstrate the theoretical connection to learning Bayesian neural networks, which proves the generalizability of the perturbed model to target domains. To enable more efficient optimization, we further employ a parameter sharing strategy, which substantially reduces the learnable parameters compared to a fully Bayesian neural network. Our model perturbation provides a new probabilistic way for domain adaptation which enables efficient adaptation to target domains while maximally preserving knowledge in source models. Experiments on several source-free benchmarks under three different evaluation settings verify the effectiveness of the proposed variational model perturbation for source-free domain adaptation.

TCS Journal 2021 Journal Article

A new construction of odd-variable rotation symmetric Boolean functions with optimal algebraic immunity and higher nonlinearity

• Sihong Su
• Jingjing Li
• Bingxin Wang

Rotation symmetric Boolean functions are potentially rich in functions of cryptographic significance. In this paper, a new construction of odd-variable rotation symmetric Boolean functions with optimal algebraic immunity is presented. By a direct calculation, the nonlinearity of the newly constructed functions is higher than the nonlinearities of all the known odd-variable rotation symmetric Boolean functions with optimal algebraic immunity. The algebraic degree and the fast algebraic immunity of our functions are also considered.

AAAI Conference 2021 Conference Paper

Balanced Open Set Domain Adaptation via Centroid Alignment

• Mengmeng Jing
• Jingjing Li
• Lei Zhu
• Zhengming Ding
• Ke Lu
• Yang Yang

Open Set Domain Adaptation (OSDA) is a challenging domain adaptation setting which allows the existence of unknown classes on the target domain. Although existing OSDA methods are good at classifying samples of known classes, they ignore the classification ability for the unknown samples, making them unbalanced OSDA methods. To alleviate this problem, we propose a balanced OSDA methods which could recognize the unknown samples while maintain high classification performance for the known samples. Specifically, to reduce the domain gaps, we first project the features to a hyperspherical latent space. In this space, we propose to bound the centroid deviation angles to not only increase the intraclass compactness but also enlarge the inter-class margins. With the bounded centroid deviation angles, we employ the statistical Extreme Value Theory to recognize the unknown samples that are misclassified into known classes. In addition, to learn better centroids, we propose an improved centroid update strategy based on sample reweighting and adaptive update rate to cooperate with centroid alignment. Experimental results on three OSDA benchmarks verify that our method can significantly outperform the compared methods and reduce the proportion of the unknown samples being misclassified into known classes.

IJCAI Conference 2021 Conference Paper

Dynamic Rebalancing Dockless Bike-Sharing System based on Station Community Discovery

• Jingjing Li
• Qiang Wang
• Wenqi Zhang
• Donghai Shi
• Zhiwei Qin

Influenced by the era of the sharing economy and mobile payment, Dockless Bike-Sharing System (Dockless BSS) is expanding in many major cities. The mobility of users constantly leads to supply and demand imbalance, which seriously affects the total profit and customer satisfaction. In this paper, we propose the Spatio-Temporal Mixed Integer Program (STMIP) with Flow-graphed Community Discovery (FCD) approach to rebalancing the system. Different from existing studies that ignore the route of trucks and adopt a centralized rebalancing, our approach considers the spatio-temporal information of trucks and discovers station communities for truck-based rebalancing. First, we propose the FCD algorithm to detect station communities. Significantly, rebalancing communities decomposes the centralized system into a distributed multi-communities system. Then, by considering the routing and velocity of trucks, we design the STMIP model with the objective of maximizing total profit, to find a repositioning policy for each station community. We design a simulator built on real-world data from DiDi Chuxing to test the algorithm performance. The extensive experimental results demonstrate that our approach outperforms in terms of service level, profit, and complexity compared with the state-of-the-art approach.

NeurIPS Conference 2021 Conference Paper

Joint Semantic Mining for Weakly Supervised RGB-D Salient Object Detection

• Jingjing Li
• Wei Ji
• Qi Bi
• Cheng Yan
• Miao Zhang
• Yongri Piao
• Huchuan Lu
• Li Cheng

Training saliency detection models with weak supervisions, e. g. , image-level tags or captions, is appealing as it removes the costly demand of per-pixel annotations. Despite the rapid progress of RGB-D saliency detection in fully-supervised setting, it however remains an unexplored territory when only weak supervision signals are available. This paper is set to tackle the problem of weakly-supervised RGB-D salient object detection. The key insight in this effort is the idea of maintaining per-pixel pseudo-labels with iterative refinements by reconciling the multimodal input signals in our joint semantic mining (JSM). Considering the large variations in the raw depth map and the lack of explicit pixel-level supervisions, we propose spatial semantic modeling (SSM) to capture saliency-specific depth cues from the raw depth and produce depth-refined pseudo-labels. Moreover, tags and captions are incorporated via a fill-in-the-blank training in our textual semantic modeling (TSM) to estimate the confidences of competing pseudo-labels. At test time, our model involves only a light-weight sub-network of the training pipeline, i. e. , it requires only an RGB image as input, thus allowing efficient inference. Extensive evaluations demonstrate the effectiveness of our approach under the weakly-supervised setting. Importantly, our method could also be adapted to work in both fully-supervised and unsupervised paradigms. In each of these scenarios, superior performance has been attained by our approach with comparing to the state-of-the-art dedicated methods. As a by-product, a CapS dataset is constructed by augmenting existing benchmark training set with additional image tags and captions.

EAAI Journal 2021 Journal Article

Local–Global Attentive Adaptation for Object Detection

• Dan Zhang
• Jingjing Li
• Xingpeng Li
• Zhekai Du
• Lin Xiong
• Mao Ye

Adversarial adaptive methods have been proven to be useful for domain transfer in many fields such as image recognition and semantic segmentation, etc However, for object detection, since each image could have different combinations of objects, brutally aligning all the images without considering their transferability may cause the notorious phenomena named ‘negative transfer’. On the other hand, strong matching the local-level features makes sense, as it not only reduces the discrepancy between different domain distributions, but preserves the category-level semantic information. However, it is hard to markedly achieve domain invariance using a simple adversarial adaptive method. In this work, we propose an effective method termed Local–Global Attentive Adaptation for object Detection (LGAAD). Our method can alleviate the negative transfer caused by improper global alignments through leveraging an adaptively and dynamically weighted transferability to highlight the more transferable images. Furthermore, the proposed method also achieves the strong matching between two domains at local-level features to alleviate the cross-domain discrepancy by using the attention mechanism after multiple local discriminators. Additionally, we also consider the domain impacts of instance-wise features and backgrounds in images with large domain divergence, a non-negligible factor for improving the domain adaptive detection model performance. Extensive experiments of various domain shift scenarios show that our method exceeds the state-of-the-art results on several public datasets. Furthermore, qualitative visualization and ablation analyzes can demonstrate the validity of our approach for attending the interested regions and instances on domain adaptation.

TIST Journal 2020 Journal Article

Flexible Multi-modal Hashing for Scalable Multimedia Retrieval

• Lei Zhu
• Xu Lu
• Zhiyong Cheng
• Jingjing Li
• Huaxiang Zhang

Multi-modal hashing methods could support efficient multimedia retrieval by combining multi-modal features for binary hash learning at the both offline training and online query stages. However, existing multi-modal methods cannot binarize the queries, when only one or part of modalities are provided. In this article, we propose a novel Flexible Multi-modal Hashing (FMH) method to address this problem. FMH learns multiple modality-specific hash codes and multi-modal collaborative hash codes simultaneously within a single model. The hash codes are flexibly generated according to the newly coming queries, which provide any one or combination of modality features. Besides, the hashing learning procedure is efficiently supervised by the pair-wise semantic matrix to enhance the discriminative capability. It could successfully avoid the challenging symmetric semantic matrix factorization and O ( n 2 ) storage cost of semantic matrix. Finally, we design a fast discrete optimization to learn hash codes directly with simple operations. Experiments validate the superiority of the proposed approach.

AAAI Conference 2020 Conference Paper

Multi-Feature Discrete Collaborative Filtering for Fast Cold-Start Recommendation

• Yang Xu
• Lei Zhu
• Zhiyong Cheng
• Jingjing Li
• Jiande Sun

Hashing is an effective technique to address the largescale recommendation problem, due to its high computation and storage efficiency on calculating the user preferences on items. However, existing hashing-based recommendation methods still suffer from two important problems: 1) Their recommendation process mainly relies on the user-item interactions and single specific content feature. When the interaction history or the content feature is unavailable (the cold-start problem), their performance will be seriously deteriorated. 2) Existing methods learn the hash codes with relaxed optimization or adopt discrete coordinate descent to directly solve binary hash codes, which results in significant quantization loss or consumes considerable computation time. In this paper, we propose a fast cold-start recommendation method, called Multi-Feature Discrete Collaborative Filtering (MFDCF), to solve these problems. Specifically, a lowrank self-weighted multi-feature fusion module is designed to adaptively project the multiple content features into binary yet informative hash codes by fully exploiting their complementarity. Additionally, we develop a fast discrete optimization algorithm to directly compute the binary hash codes with simple operations. Experiments on two public recommendation datasets demonstrate that MFDCF outperforms the stateof-the-arts on various aspects.

NeurIPS Conference 2020 Conference Paper

Unsupervised Text Generation by Learning from Search

• Jingjing Li
• Zichao Li
• Lili Mou
• Xin Jiang
• Michael Lyu
• Irwin King

In this work, we propose TGLS, a novel framework for unsupervised Text Generation by Learning from Search. We start by applying a strong search algorithm (in particular, simulated annealing) towards a heuristically defined objective that (roughly) estimates the quality of sentences. Then, a conditional generative model learns from the search results, and meanwhile smooth out the noise of search. The alternation between search and learning can be repeated for performance bootstrapping. We demonstrate the effectiveness of TGLS on two real-world natural language generation tasks, unsupervised paraphrasing and text formalization. Our model significantly outperforms unsupervised baseline methods in both tasks. Especially, it achieves comparable performance to strong supervised methods for paraphrase generation.

AAAI Conference 2019 Conference Paper

From Zero-Shot Learning to Cold-Start Recommendation

• Jingjing Li
• Mengmeng Jing
• Ke Lu
• Lei Zhu
• Yang Yang
• Zi Huang

Zero-shot learning (ZSL) and cold-start recommendation (CSR) are two challenging problems in computer vision and recommender system, respectively. In general, they are independently investigated in different communities. This paper, however, reveals that ZSL and CSR are two extensions of the same intension. Both of them, for instance, attempt to predict unseen classes and involve two spaces, one for direct feature representation and the other for supplementary description. Yet there is no existing approach which addresses CSR from the ZSL perspective. This work, for the first time, formulates CSR as a ZSL problem, and a tailor-made ZSL method is proposed to handle CSR. Specifically, we propose a Lowrank Linear Auto-Encoder (LLAE), which challenges three cruxes, i. e. , domain shift, spurious correlations and computing efficiency, in this paper. LLAE consists of two parts, a low-rank encoder maps user behavior into user attributes and a symmetric decoder reconstructs user behavior from user attributes. Extensive experiments on both ZSL and CSR tasks verify that the proposed method is a win-win formulation, i. e. , not only can CSR be handled by ZSL models with a significant performance improvement compared with several conventional state-of-the-art methods, but the consideration of CSR can benefit ZSL as well.

NeurIPS Conference 2019 Conference Paper

Memory-oriented Decoder for Light Field Salient Object Detection

• Miao Zhang
• Jingjing Li
• JI WEI
• Yongri Piao
• Huchuan Lu

Light field data have been demonstrated in favor of many tasks in computer vision, but existing works about light field saliency detection still rely on hand-crafted features. In this paper, we present a deep-learning-based method where a novel memory-oriented decoder is tailored for light field saliency detection. Our goal is to deeply explore and comprehensively exploit internal correlation of focal slices for accurate prediction by designing feature fusion and integration mechanisms. The success of our method is demonstrated by achieving the state of the art on three datasets. We present this problem in a way that is accessible to members of the community and provide a large-scale light field dataset that facilitates comparisons across algorithms. The code and dataset will be made publicly available.

AAAI Conference 2019 Conference Paper

MR-NET: Exploiting Mutual Relation for Visual Relationship Detection

• Yi Bin
• Yang Yang
• Chaofan Tao
• Zi Huang
• Jingjing Li
• Heng Tao Shen

Inferring the interactions between objects, a. k. a visual relationship detection, is a crucial point for vision understanding, which captures more definite concepts than object detection. Most previous work that treats the interaction between a pair of objects as a one way fail to exploit the mutual relation between objects, which is essential to modern visual application. In this work, we propose a mutual relation net, dubbed MR-Net, to explore the mutual relation between paired objects for visual relationship detection. Specifically, we construct a mutual relation space to model the mutual interaction of paired objects, and employ linear constraint to optimize the mutual interaction, which is called mutual relation learning. Our mutual relation learning does not introduce any parameters, and can adapt to improve the performance of other methods. In addition, we devise a semantic ranking loss to discriminatively penalize predicates with semantic similarity, which is ignored by traditional loss function (e. g. , cross entropy with softmax). Then, our MR-Net optimizes the mutual relation learning together with semantic ranking loss with a siamese network. The experimental results on two commonly used datasets (VG and VRD) demonstrate the superior performance of the proposed approach.

IJCAI Conference 2018 Conference Paper

Adaptive Collaborative Similarity Learning for Unsupervised Multi-view Feature Selection

• Xiao Dong
• Lei Zhu
• Xuemeng Song
• Jingjing Li
• Zhiyong Cheng

In this paper, we investigate the research problem of unsupervised multi-view feature selection. Conventional solutions first simply combine multiple pre-constructed view-specific similarity structures into a collaborative similarity structure, and then perform the subsequent feature selection. These two processes are separate and independent. The collaborative similarity structure remains fixed during feature selection. Further, the simple undirected view combination may adversely reduce the reliability of the ultimate similarity structure for feature selection, as the view-specific similarity structures generally involve noises and outlying entries. To alleviate these problems, we propose an adaptive collaborative similarity learning (ACSL) for multi-view feature selection. We propose to dynamically learn the collaborative similarity structure, and further integrate it with the ultimate feature selection into a unified framework. Moreover, a reasonable rank constraint is devised to adaptively learn an ideal collaborative similarity structure with proper similarity combination weights and desirable neighbor assignment, both of which could positively facilitate the feature selection. An effective solution guaranteed with the proved convergence is derived to iteratively tackle the formulated optimization problem. Experiments demonstrate the superiority of the proposed approach.

IJCAI Conference 2018 Conference Paper

Fine-grained Image Classification by Visual-Semantic Embedding

• Huapeng Xu
• Guilin Qi
• Jingjing Li
• Meng Wang
• Kang Xu
• Huan Gao

This paper investigates a challenging problem, which is known as fine-grained image classification(FGIC). Different from conventional computer visionproblems, FGIC suffers from the large intraclassdiversities and subtle inter-class differences. Existing FGIC approaches are limited to exploreonly the visual information embedded in the images. In this paper, we present a novel approachwhich can use handy prior knowledge from eitherstructured knowledge bases or unstructured text tofacilitate FGIC. Specifically, we propose a visual-semanticembedding model which explores semanticembedding from knowledge bases and text, andfurther trains a novel end-to-end CNN frameworkto linearly map image features to a rich semanticembedding space. Experimental results on a challenginglarge-scale UCSD Bird-200-2011 datasetverify that our approach outperforms several state-of-the-art methods with significant advances.

IJCAI Conference 2016 Conference Paper

Joint Feature Selection and Structure Preservation for Domain Adaptation

• Jingjing Li
• Jidong Zhao
• Ke Lu

The essence of domain adaptation is to explore common latent factors shared by the involved domains. These factors can be specific features or geometric structures. Most of previous methods exploit either the shared features or the shared geometric structures separately. However, the two strategies are complementary with each other and jointly exploring them is more optimal. This paper proposes a novel approach, named joint Feature Selection and Structure Preservation (FSSP), for unsupervised domain adaptation. FSSP smoothly integrates structure preservation and feature selection into a unified optimization problem. Intensive experiments on text categorization, image classification and video event recognition demonstrate that our method performs better, even with up to 30% improvement in average, compared with the state-of-the-art methods.