Author name cluster

Ye Liu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

31 papers

2 author rows

AAAI Conference 2026 Conference Paper

Graph Contrastive Learning with Balanced Hard Negatives and Fine-grained Semantic-aware Positives

Hongshan Pu
Haoxu Zhang
Ye Liu
Hongmin Cai

Graph contrastive learning (GCL) aims to learn representations by bringing semantically similar graphs closer and pushing dissimilar ones farther apart without label supervision. Hard negatives, which refer to graphs that have different labels but similar embeddings to the target graph, play a key role in improving representation discrimination. However, current methods that generate both high-quality positives and hard negatives face two challenges: (1) Hard negative sample generation often suffers from class imbalance, resulting in unequal attention across classes and reduced discriminative power in the learned representations. (2) The typical binary positive sample generation approach, which divides the graph into important and unimportant semantic regions, overlooks regions that negatively impact semantics and mislead model predictions. To address these issues, we introduce a novel method named BalanceGCL, which enhance graph contrastive learning with balanced hard negatives and fine-grained semantic-aware positives. BalanceGCL comprises two modules: Balanced Hard Negative graphs generation (BHN) and Fine-grained Semantic-aware Positive graphs generation (FSP). Inspired by the counterfactual mechanism, BHN generates balanced hard negatives that remain structurally similar to the original graph while inducing a controlled semantic shift. To ensure class balance, BHN iteratively constructs one hard negative sample for each class, ensuring an even distribution of negative samples across all alternative categories. FSP leverages the semantic differences between original graphs and balanced hard negatives to identify positively contributing, negatively contributing, and unimportant regions. By enhancing the influence of positive contributors, suppressing negative ones, and perturbing unimportant areas, it generates more reliable and semantically complete positive samples. The proposed method outperforms state-of-the-art GCL techniques across 14 datasets in graph classification and transfer learning tasks, demonstrating its effectiveness in tackling class imbalance and identifying fine-grained semantic-aware regions.

PDF Details DOI

JBHI Journal 2026 Journal Article

GraphSTAR: Proximal Operator-Based Graph Neural Network Enhanced by Dynamic Graph Aggregation for Spatial Transcriptomics

Junyu Li
Jingquan Yan
Yi Liao
Wenxiong Liao
Ye Liu
Hongmin Cai

Spatial transcriptomics technologies carry out advanced sequencing analysis of molecular profiles with a spatial context, providing multi-source information essential for elucidating biological regulatory mechanisms. Nonetheless, it poses challenges in the integration of raw spatial coordinates with high-dimensional gene expression profiles in their native feature space. While spatial-aware methods effectively aggregate molecular information from local spatial neighborhoods, they fail to explore the long-range relationships associated with gene expression data. To address this issue, this paper introduces a novel approach termed GraphSTAR that encodes both spatial and gene expression data into undirected graphs, characterizing the local spatial proximity and global transcriptional similarity, respectively. Through a graph aggregation process, GraphSTAR integrates these diverse data sources within a joint graph structure, effectively modeling both local neighborhood relationships and long-range functional associations. Subsequently, a reassembled graph neural network is established by incorporating the graph aggregation into the feed-forward propagation using proximal operators, progressively refining spatial-informed latent representation to decipher spatial expression patterns of genes. Extensive experiments on benchmark datasets demonstrate that GraphSTAR outperforms state-of-the-art methods in both spatial domain identification and cell-type annotation tasks.

Details DOI

AAAI Conference 2026 Conference Paper

MIGDiff: Multi-attributes Imputations for Attribute-missing Graphs via Graph Denoising Diffusion Model

Ye Liu
Yang Chen
Hongmin Cai

The missing of graph attributes poses a significant challenge in graph representation learning. Some existing graph attribute completion methods adopt the shared-space hypothesis or employ end-to-end frameworks to perform single-attribute imputation. However, these models can only generate one single attribute with a few specific patterns that either adhere to prior knowledge or are optimal for downstream tasks, making it difficult to capture the full range of variations in the target attribute distribution. This limitation negatively impacts the model's generalizability and efficiency. Therefore, to address this issue, we proposed a new method based on a graph denoising diffusion model, called Multi-attribute Imputation Graph Denoising Diffusion Model (MIGDiff), which can generate multiple high-quality attributes. Specifically, it employs a Dual-source Auto-encoder on existing attributes and graph topology to extract reliable knowledge, which serves as a condition for training the diffusion module. Within diffusion, noise is added to the structural embeddings of nodes without attributes in the forward process. In the reverse process, a Structure-aware Denoising Network is devised to integrate feature and structural information via an attention mechanism and to perform neighbor-guided refinement based on graph connectivity, thereby enhancing denoising and accurately recovering missing attributes while effectively maintaining structural consistency and distributional fidelity. During generation, multiple initial values are sampled to produce diverse attribute imputations, avoiding focusing on a few easy-to-learn patterns. Extensive experiments conducted on four public datasets highlight the state-of-the-art performance of MIGDiff in both attribute imputation and node classification tasks.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Perceive More with Less: LiDAR Point Cloud Compression at Just Recognizable Distortion for 3D Scene Understanding

Miaohui Wang
Runnan Huang
Taojun Liu
Shuyuan Lin
Ye Liu
Yun Song

Existing LiDAR point cloud (LPC) data coding methods primarily focus on balancing compression efficiency and reconstruction quality according to the human vision system (HVS). However, these methods rarely consider the requirements of downstream scene understanding tasks from the perspective of the machine vision system (MVS). To address this challenge, we explore the maximum degree of LPC compression that has negligible impact on perception accuracy, called LPC-based just recognizable compression distortion (lpcJRCD). Specifically, we introduce a novel point-wise quantization approach for constructing a MVS-based LiDAR dataset and present a new lpcJRCD-guided intelligent compression framework tailored for MVS applications. To enhance MVS-based LPC compression efficiency, we develop a dual-feature interaction (DFI) module that fuses point and voxel features. Additionally, we propose a mask-based loss function to ensure accurate point-wise quality level prediction. Experimental results demonstrate the effectiveness of our proposed model in reducing the average bit rate by up to 94.98% while preserving perception accuracy in autonomous vehicles.

PDF Details DOI

AAAI Conference 2026 Conference Paper

The Last Byte: Learning Just Enough for Machine-Oriented Image Compression

Wuyuan Xie
Zhenming Li
Ye Liu
Jian Jin
Yun Song
Miaohui Wang

Just recognizable distortion (JRD) has been introduced for image compression for machines, aiming to quantify the maximum coding distortion that can be tolerated by a specific perception model, thereby defining the upper bound of machine vision redundancy (MVR). However, existing JRD-based redundancy estimation methods face three key challenges: limited dataset annotation accuracy, low prediction efficiency, and insufficient perception accuracy, all of which hinder their practical deployment. To address these limitations, we propose a new MVR-Net, a frame-wise efficient JRD prediction method that generates the optimal encoding quantization map in a single inference pass. Furthermore, we refine the annotation standard for JRD datasets based on experimental insights, enhancing the precision of recognizable redundancy measurement. Compared to stateof-the-art methods, MVR-Net achieves a superior balance between bitrate reduction and perception accuracy in JRD-guided compression, while offering up to a 40,000× speed improvement, demonstrating its practicality and efficiency for real-world applications.

PDF Details DOI

TMLR Journal 2026 Journal Article

VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents

Rui Meng
Ziyan Jiang
Ye Liu
Mingyi Su
Xinyi Yang
Yuepeng Fu
Can Qin
Raghuveer Thirukovalluru

Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, retrieval-augmented generation (RAG) systems, and recommendation. To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering -- spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings.

PDF Details

AAAI Conference 2025 Conference Paper

AFFAKT: A Hierarchical Optimal Transport Based Method for Affective Facial Knowledge Transfer in Video Deception Detection

Zihan Ji
Xuetao Tian
Ye Liu

The scarcity of high-quality large-scale labeled datasets poses a huge challenge for employing deep learning models in video deception detection. To address this issue, inspired by the psychological theory on the relation between deception and expressions, we propose a novel method called AFFAKT in this paper, which enhances the classification performance by transferring useful and correlated knowledge from a large facial expression dataset. Two key challenges in knowledge transfer arise: 1) how much knowledge of facial expression data should be transferred and 2) how to effectively leverage transferred knowledge for the deception classification model during inference. Specifically, the optimal relation mapping between facial expression classes and deception samples is firstly quantified using proposed H-OTKT module and then transfers knowledge from the facial expression dataset to deception samples. Moreover, a correlation prototype within another proposed module SRKB is well designed to retain the invariant correlations between facial expression classes and deception classes through momentum updating. During inference, the transferred knowledge is fine-tuned with the correlation prototype using a sample-specific re-weighting strategy. Experimental results on two deception detection datasets demonstrate the superior performance of our proposed method. The interpretability study reveals high associations between deception and negative affections, which coincides with the theory in psychology.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining

Raghuveer Thirukovalluru
Rui Meng
Ye Liu
Karthikeyan K
Mingyi Su
Ping Nie
Semih Yavuz
Yingbo Zhou

Contrastive learning (CL) is a prevalent technique for training embedding models, which pulls semantically similar examples (positives) closer in the representation space while pushing dissimilar ones (negatives) further apart. A key source of negatives are "in-batch" examples, i. e. , positives from other examples in the batch. Effectiveness of such models is hence strongly influenced by the size and quality of training batches. In this work, we propose Breaking the Batch Barrier (B3), a novel batch construction strategy designed to curate high-quality batches for CL. Our approach begins by using a pretrained teacher embedding model to rank all examples in the dataset, from which a sparse similarity graph is constructed. A community detection algorithm is then applied to this graph to identify clusters of examples that serve as strong negatives for one another. The clusters are then used to construct batches that are rich in in-batch negatives. Empirical results on the MMEB multimodal embedding benchmark (36 tasks) demonstrate that our method sets a new state of the art, outperforming previous best methods by +1. 3 and +2. 9 points at the 7B and 2B model scales, respectively. Notably, models trained with B3 surpass existing state-of-the-art results even with a batch size as small as 64, which is 4–16× smaller than that required by other methods. Moreover, experiments show that B3 generalizes well across domains and tasks, maintaining strong performance even when trained with considerably weaker teachers.

PDF Details

ICML Conference 2025 Conference Paper

DragSolver: A Multi-Scale Transformer for Real-World Automotive Drag Coefficient Estimation

Ye Liu
Yuntian Chen

Automotive drag coefficient ($C_d$) is pivotal to energy efficiency, fuel consumption, and aerodynamic performance. However, costly computational fluid dynamics (CFD) simulations and wind tunnel tests struggle to meet the rapid-iteration demands of automotive design. We present DragSolver, a Transformer-based framework for rapid and accurate $C_d$ estimation from large-scale, diverse 3D vehicle models. DragSolver tackles four key real-world challenges: (1) multi-scale feature extraction to capture both global shape and fine local geometry; (2) heterogeneous scale normalization to handle meshes with varying sizes and densities; (3) surface-guided gating to suppress internal structures irrelevant to external aerodynamics; and (4) epistemic uncertainty estimation via Monte Carlo dropout for risk-aware design. Extensive evaluations on three industrial-scale datasets (DrivaerNet, DrivaerNet++, and DrivaerML) show that DragSolver outperforms existing approaches in accuracy and generalization, achieving an average reduction of relative $L_2$ error by 58. 7% across real-world datasets. Crucially, DragSolver is the first to achieve reliable, real-time $C_d$ inference on production-level automotive geometries.

Details

NeurIPS Conference 2025 Conference Paper

Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation

Fei Wang
Li Shen
Liang Ding
Chao Xue
Ye Liu
Changxing Ding

Large Language Models (LLMs) excel at natural language processing tasks, but their massive size leads to high computational and storage demands. Recent works have sought to reduce their model size through layer-wise structured pruning. However, they tend to ignore retaining the capabilities in the pruned part. In this work, we re-examine structured pruning paradigms and uncover several key limitations: 1) notable performance degradation due to direct layer removal, 2) incompetent linear weighted layer aggregation, and 3) the lack of effective post-training recovery mechanisms. To address these limitations, we propose CoMe, including a progressive layer pruning framework with a Concatenation-based Merging technology and a hierarchical distillation post-training process. Specifically, we introduce a channel sensitivity metric that utilizes activation intensity and weight norms for fine-grained channel selection. Subsequently, we employ a concatenation-based layer merging method to fuse the most critical channels in the adjacent layers, enabling a progressive model size reduction. Finally, we propose a hierarchical distillation protocol, which leverages the correspondences between the original and pruned model layers established during pruning, enabling efficient knowledge transfer. Experiments on seven benchmarks show that CoMe achieves state-of-the-art performance; when pruning 30% of LLaMA-2-7b's parameters, the pruned model retains 83% of its original average accuracy.

PDF Details

AAAI Conference 2025 Conference Paper

mmFAS: Multimodal Face Anti-Spoofing Using Multi-Level Alignment and Switch-Attention Fusion

Geng Chen
Wuyuan Xie
Di Lin
Ye Liu
Miaohui Wang

The increasing number of presentation attacks on reliable face matching has raised concerns and garnered attention towards face anti-spoofing (FAS). However, existing methods for FAS modeling commonly fuse multiple visual modalities (e.g., RGB, Depth, and Infrared) in a straightforward manner, disregarding latent feature gaps that can hinder representation learning. To address this challenge, we propose a novel multimodal FAS framework (mmFAS) that focuses on explicit alignment and fusion of latent features across different modalities. Specifically, we develop a multimodal alignment module to alleviate the latent feature gap by using instance-level contrastive learning and class-level matching simultaneously. Further, we explore a new switch-attention based fusion module to automatically aggregate complementary information and control model complexity. To evaluate the anti-spoofing performance more effectively, we adopt a challenging yet meaningful cross-database protocol involving four benchmark multimodal FAS datasets to simulate realworld scenarios. Extensive experimental results demonstrate the effectiveness of mmFAS in improving the accuracy of FAS systems, outperforming 10 representative methods.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

Plug-and-play Feature Causality Decomposition for Multimodal Representation Learning

Ye Liu
Zihan Ji
Hongmin Cai

Multimodal representation learning is critical for a wide range of applications, such as multimodal sentiment analysis. Current multimodal representation learning methods mainly focus on the multimodal alignment or fusion strategies, such that the complementary and consistent information among heterogeneous modalities can be fully explored. However, they mistakenly treat the uncertainty noise within each modality as the complementary information, failing to simultaneously leverage both consistent and complementary information while eliminating the aleatoric uncertainty within each modality. To address this issue, we propose a plug-and-play feature causality decomposition method for multimodal representation learning from causality perspective, which can be integrated into existing models with no affects on the original model structures. Specifically, to deal with the heterogeneity and consistency, according to whether it can be aligned with other modalities, the unimodal feature is first disentangled into two parts: modality-invariant (the synergistic information shared by all heterogeneous modalities) and modality-specific part. To deal with complementarity and uncertainty, the modality-specific part is further decomposed into unique and redundant features, where the redundant feature is removed and the unique feature is reserved based on the backdoor-adjustment. The effectiveness of noise removal is supported by causality theory. Finally, the task-related information, including both synergistic and unique components, is further fed to the original fusion module to obtain the final multimodal representations. Extensive experiments show the effectiveness of our proposed strategies.

PDF Details

NeurIPS Conference 2025 Conference Paper

UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning

Ye Liu
Zongyang Ma
Junfu Pu
Zhongang Qi
Yang Wu
Ying Shan
Chang Chen

Recent advances in Large Multi-modal Models (LMMs) have demonstrated their remarkable success as general-purpose multi-modal assistants, with particular focuses on holistic image- and video-language understanding. Conversely, less attention has been given to scaling fine-grained pixel-level understanding capabilities, where the models are expected to realize pixel-level alignment between visual signals and language semantics. Some previous studies have applied LMMs to related tasks such as region-level captioning and referring expression segmentation. However, these models are limited to performing either referring or segmentation tasks independently and fail to integrate these fine-grained perception capabilities into visual reasoning. To bridge this gap, we propose UniPixel, a large multi-modal model capable of flexibly comprehending visual prompt inputs and generating mask-grounded responses. Our model distinguishes itself by seamlessly integrating pixel-level perception with general visual understanding capabilities. Specifically, UniPixel processes visual prompts and generates relevant masks on demand, and performs subsequent reasoning conditioning on these intermediate pointers during inference, thereby enabling fine-grained pixel-level reasoning. The effectiveness of our approach has been verified on 10 benchmarks across a diverse set of tasks, including pixel-level referring/segmentation and object-centric understanding in images/videos. A novel PixelQA task that jointly requires referring, segmentation, and question answering is also designed to verify the flexibility of our method.

PDF Details

EAAI Journal 2024 Journal Article

A fusion scheme for eliminating input interference induced by spelling errors

Chi Wei
Shaobin Huang
Rongsheng Li
Ye Liu
Naiyu Yan

Spelling errors are a prevalent form of interference that may result from human handwriting or natural language processing systems, such as optical character recognition. This type of interference presents a challenge for natural language processing systems that rely on context-based embeddings to comprehend the exact semantics of a sentence. The proposed fusion scheme is the main focus of this paper, which aims to correct spelling errors in sentences. The scheme employs both the original input and the masked input. The masked input is acquired by utilizing a detection module to mask any spelling errors present in the original input. By utilizing the masked input, the scheme is capable of learning the real sentence semantic information that is not affected by spelling errors. Utilizing the original input can prevent the loss of input information resulting from masking operations. Finally, the utilization of the Kullback-Leibler divergence is employed to establish concordance between the two output distributions within the framework, consequently facilitating the acquisition of comprehensive input information, all while mitigating the influence of spelling errors. Experiments were conducted on two widely used benchmarks. Compared with previous methods, the approach presented herein exhibited superior performance as it achieved improvements on two benchmarks.

Details DOI

NeurIPS Conference 2024 Conference Paper

E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding

Ye Liu
Zongyang Ma
Zhongang Qi
Yang Wu
Ying Shan
Chang W. Chen

Recent advances in Video Large Language Models (Video-LLMs) have demonstrated their great potential in general-purpose video understanding. To verify the significance of these models, a number of benchmarks have been proposed to diagnose their capabilities in different scenarios. However, existing benchmarks merely evaluate models through video-level question-answering, lacking fine-grained event-level assessment and task diversity. To fill this gap, we introduce E. T. Bench (Event-Level & Time-Sensitive Video Understanding Benchmark), a large-scale and high-quality benchmark for open-ended event-level video understanding. Categorized within a 3-level task taxonomy, E. T. Bench encompasses 7. 3K samples under 12 tasks with 7K videos (251. 4h total length) under 8 domains, providing comprehensive evaluations. We extensively evaluated 8 Image-LLMs and 12 Video-LLMs on our benchmark, and the results reveal that state-of-the-art models for coarse-level (video-level) understanding struggle to solve our fine-grained tasks, e. g. , grounding event-of-interests within videos, largely due to the short video context length, improper time representations, and lack of multi-event training data. Focusing on these issues, we further propose a strong baseline model, E. T. Chat, together with an instruction-tuning dataset E. T. Instruct 164K tailored for fine-grained event-level understanding. Our simple but effective solution demonstrates superior performance in multiple scenarios.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition

Mengke Li
Ye Liu
Yang Lu
Yiqun Zhang
Yiu-ming Cheung
Hui Huang

Long-tailed visual recognition has received increasing attention recently. Despite fine-tuning techniques represented by visual prompt tuning (VPT) achieving substantial performance improvement by leveraging pre-trained knowledge, models still exhibit unsatisfactory generalization performance on tail classes. To address this issue, we propose a novel optimization strategy called Gaussian neighborhood minimization prompt tuning (GNM-PT), for VPT to address the long-tail learning problem. We introduce a novel Gaussian neighborhood loss, which provides a tight upper bound on the loss function of data distribution, facilitating a flattened loss landscape correlated to improved model generalization. Specifically, GNM-PT seeks the gradient descent direction within a random parameter neighborhood, independent of input samples, during each gradient update. Ultimately, GNM-PT enhances generalization across all classes while simultaneously reducing computational overhead. The proposed GNM-PT achieves state-of-the-art classification accuracies of 90. 3%, 76. 5%, and 50. 1% on benchmark datasets CIFAR100-LT (IR 100), iNaturalist 2018, and Places-LT, respectively. The source code is available at https: //github. com/Keke921/GNM-PT.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs

Jihyung Kil
Zheda Mai
Justin Lee
Arpita Chowdhury
Zihe Wang
Kerrie Cheng
Lemeng Wang
Ye Liu

The ability to compare objects, scenes, or situations is crucial for effective decision-making and problem-solving in everyday life. For instance, comparing the freshness of apples enables better choices during grocery shopping, while comparing sofa designs helps optimize the aesthetics of our living space. Despite its significance, the comparative capability is largely unexplored in artificial general intelligence (AGI). In this paper, we introduce MLLM-CompBench, a benchmark designed to evaluate the comparative reasoning capability of multimodal large language models (MLLMs). MLLM-CompBench mines and pairs images through visually oriented questions covering eight dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. We curate a collection of around 40K image pairs using metadata from diverse vision datasets and CLIP similarity scores. These image pairs span a broad array of visual domains, including animals, fashion, sports, and both outdoor and indoor scenes. The questions are carefully crafted to discern relative characteristics between two images and are labeled by human annotators for accuracy and relevance. We use MLLM-CompBench to evaluate recent MLLMs, including GPT-4V(ision), Gemini-Pro, and LLaVA-1. 6. Our results reveal notable shortcomings in their comparative abilities. We believe MLLM-CompBench not only sheds light on these limitations but also establishes a solid foundation for future enhancements in the comparative capability of MLLMs.

PDF Details DOI

YNIMG Journal 2023 Journal Article

Decoding the temporal representation of facial expression in face-selective regions

Zhihao Zhang
Tong Chen
Ye Liu
Chongyang Wang
Ke Zhao
Chang Hong Liu
Xiaolan Fu

The ability of humans to discern facial expressions in a timely manner typically relies on distributed face-selective regions for rapid neural computations. To study the time course in regions of interest for this process, we used magnetoencephalography (MEG) to measure neural responses participants viewed facial expressions depicting seven types of emotions (happiness, sadness, anger, disgust, fear, surprise, and neutral). Analysis of the time-resolved decoding of neural responses in face-selective sources within the inferior parietal cortex (IP-faces), lateral occipital cortex (LO-faces), fusiform gyrus (FG-faces), and posterior superior temporal sulcus (pSTS-faces) revealed that facial expressions were successfully classified starting from ∼100 to 150 ms after stimulus onset. Interestingly, the LO-faces and IP-faces showed greater accuracy than FG-faces and pSTS-faces. To examine the nature of the information processed in these face-selective regions, we entered with facial expression stimuli into a convolutional neural network (CNN) to perform similarity analyses against human neural responses. The results showed that neural responses in the LO-faces and IP-faces, starting ∼100 ms after the stimuli, were more strongly correlated with deep representations of emotional categories than with image level information from the input images. Additionally, we observed a relationship between the behavioral performance and the neural responses in the LO-faces and IP-faces, but not in the FG-faces and lpSTS-faces. Together, these results provided a comprehensive picture of the time course and nature of information involved in facial expression discrimination across multiple face-selective regions, which advances our understanding of how the human brain processes facial expressions.

Details DOI

AAAI Conference 2023 Conference Paper

Just Noticeable Visual Redundancy Forecasting: A Deep Multimodal-Driven Approach

Wuyuan Xie
Shukang Wang
Sukun Tian
Lirong Huang
Ye Liu
Miaohui Wang

Just noticeable difference (JND) refers to the maximum visual change that human eyes cannot perceive, and it has a wide range of applications in multimedia systems. However, most existing JND approaches only focus on a single modality, and rarely consider the complementary effects of multimodal information. In this article, we investigate the JND modeling from an end-to-end homologous multimodal perspective, namely hmJND-Net. Specifically, we explore three important visually sensitive modalities, including saliency, depth, and segmentation. To better utilize homologous multimodal information, we establish an effective fusion method via summation enhancement and subtractive offset, and align homologous multimodal features based on a self-attention driven encoder-decoder paradigm. Extensive experimental results on eight different benchmark datasets validate the superiority of our hmJND-Net over eight representative methods.

PDF Details DOI

IJCAI Conference 2023 Conference Paper

Keep Skills in Mind: Understanding and Implementing Skills in Commonsense Question Answering

Meikai Bao
Qi Liu
Kai Zhang
Ye Liu
Linan Yue
Longfei Li
Jun Zhou

Commonsense Question Answering (CQA) aims to answer questions that require human commonsense. Closed-book CQA, as one of the subtasks, requires the model to answer questions without retrieving external knowledge, which emphasizes the importance of the model's problem-solving ability. Most previous methods relied on large-scale pre-trained models to generate question-related knowledge while ignoring the crucial role of skills in the process of answering commonsense questions. Generally, skills refer to the learned ability in performing a specific task or activity, which are derived from knowledge and experience. In this paper, we introduce a new approach named Dynamic Skill-aware Commonsense Question Answering (DSCQA), which transcends the limitations of traditional methods by informing the model about the need for each skill in questions and utilizes skills as a critical driver in CQA process. To be specific, DSCQA first employs commonsense skill extraction module to generate various skill representations. Then, DSCQA utilizes dynamic skill module to generate dynamic skill representations. Finally, in perception and emphasis module, various skills and dynamic skill representations are used to help question-answering process. Experimental results on two publicly available CQA datasets show the effectiveness of our proposed model and the considerable impact of introducing skills.

PDF Details DOI

YNICL Journal 2022 Journal Article

Effects of acute high intraocular pressure on red-green and blue-yellow cortical color responses in non-human primates

Mengwei Li
Xiaoxiao Chen
Nini Yuan
Yiliang Lu
Ye Liu
Hongliang Gong
Liling Qian
Ian Max Andolina

Glaucoma is a leading cause of irreversible blindness worldwide, and intraocular pressure (IOP) is an established and modifiable risk factor for both chronic and acute glaucoma. The relationship between color vision deficits and chronic glaucoma has been described previously. However, the effects of acute glaucoma or acute primary angle closure, which has high prevalence in China, on color vision remains unclear. To address the above question, red-green or blue-yellow color responses in V1, V2, and V4 of seven rhesus macaques were monitored using intrinsic-signal optical imaging while monocular anterior chamber perfusions were performed to reversibly elevate IOP acutely over a clinically observed range of 30 to 90 mmHg. We found that the cortical population responses to both red-green and blue-yellow grating stimuli, systematically decreased as IOP increased from 30 to 90 mmHg. Although a similar decrement in magnitude was noted in V1, V2, and V4, blue-yellow responses were consistently more impaired than red-green responses at all levels of acute IOP elevation and in all monitored visual areas. This physiological study in non-human primates demonstrates that acute IOP elevations substantially depress the ability of the visual cortex to register color information. This effect is more severe for blue-yellow responses than for red-green responses, suggesting selective impairment of the koniocellular pathways compared with the parvocellular pathways. Together, we infer that blue-yellow color vision might be the most vulnerable visual function in acute glaucoma patients.

Details DOI

IJCAI Conference 2021 Conference Paper

Concept extraction aims to find words or phrases describing a concept from massive texts. Recently, researchers propose many neural network-based methods to automatically extract concepts. Although these methods for this task show promising results, they ignore structured information in the raw textual data (e. g. , title, topic, and clue words). In this paper, we propose a novel model, named Guided Attention Concept Extraction Network (GACEN), which uses title, topic, and clue words as additional supervision to provide guidance directly. Specifically, GACEN comprises two attention networks, one of them is to gather the relevant title and topic information for each context word in the document. The other one aims to model the implicit connection between informative words (clue words) and concepts. Finally, we aggregate information from two networks as input to Conditional Random Field (CRF) to model dependencies in the output. We collected clue words for three well-studied datasets. Extensive experiments demonstrate that our model outperforms the baseline models with a large margin, especially when the labeled data is insufficient.

PDF Details DOI

AAAI Conference 2021 Conference Paper

KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning

Ye Liu
Yao Wan
Lifang He
Hao Peng
Philip S. Yu

Generative commonsense reasoning which aims to empower machines to generate sentences with the capacity of reasoning over a set of concepts is a critical bottleneck for text generation. Even the state-of-the-art pre-trained language generation models struggle at this task and often produce implausible and anomalous sentences. One reason is that they rarely consider incorporating the knowledge graph which can provide rich relational information among the commonsense concepts. To promote the ability of commonsense reasoning for text generation, we propose a novel knowledge graphaugmented pre-trained language generation model KG-BART, which encompasses the complex relations of concepts through the knowledge graph and produces more logical and natural sentences as output. Moreover, KG-BART can leverage the graph attention to aggregate the rich concept semantics that enhances the model generalization on unseen concept sets. Experiments on benchmark CommonGen dataset verify the effectiveness of our proposed approach by comparing with several strong pre-trained language generation models, particularly KG-BART outperforms BART by 5. 80, 4. 60, in terms of BLEU-3, 4. Moreover, we also show that the generated context by our model can work as background scenarios to benefit downstream commonsense QA tasks. 1

PDF Details

ECAI Conference 2020 Conference Paper

Document and Word Representations Generated by Graph Convolutional Network and BERT for Short Text Classification

Zhihao Ye
Gongyao Jiang
Ye Liu
Zhiyong Li 0001
Jin Yuan 0002

In many studies, the graph convolution neural networks were used to solve different natural language processing (NLP) problems. However, few researches employ graph convolutional network for text classification, especially for short text classification. In this work, a special text graph of the short-text corpus is created, and then a short-text graph convolutional network (STGCN) is developed. Specifically, different topic models for short text are employed, and a short text short-text graph based on the word co-occurrence, document word relations, and text topic information, is developed. The word and sentence representations generated by the STGCN are considered as the classification feature. In addition, a pre-trained word vector obtained by the BERTs hidden layer is employed, which greatly improves the classification effect of our model. The experimental results show that our model outperforms the state-of-the-art models on multiple short text datasets.

Details

IJCAI Conference 2019 Conference Paper

Learning Multi-Objective Rewards and User Utility Function in Contextual Bandits for Personalized Ranking

Nirandika Wanigasekara
Yuxuan Liang
Siong Thye Goh
Ye Liu
Joseph Jay Williams
David S. Rosenblum

This paper tackles the problem of providing users with ranked lists of relevant search results, by incorporating contextual features of the users and search results, and learning how a user values multiple objectives. For example, to recommend a ranked list of hotels, an algorithm must learn which hotels are the right price for users, as well as how users vary in their weighting of price against the location. In our paper, we formulate the context-aware, multi-objective, ranking problem as a Multi-Objective Contextual Ranked Bandit (MOCR-B). To solve the MOCR-B problem, we present a novel algorithm, named Multi-Objective Utility-Upper Confidence Bound (MOU-UCB). The goal of MOU-UCB is to learn how to generate a ranked list of resources that maximizes the rewards in multiple objectives to give relevant search results. Our algorithm learns to predict rewards in multiple objectives based on contextual information (combining the Upper Confidence Bound algorithm for multi-armed contextual bandits with neural network embeddings), as well as learns how a user weights the multiple objectives. Our empirical results reveal that the ranked lists generated by MOU-UCB lead to better click-through rates, compared to approaches that do not learn the utility function over multiple reward objectives.

PDF Details

AAAI Conference 2018 Conference Paper

Multi-View Multi-Graph Embedding for Brain Network Clustering Analysis

Ye Liu
Lifang He
Bokai Cao
Philip Yu
Ann Ragin
Alex Leow

Network analysis of human brain connectivity is critically important for understanding brain function and disease states. Embedding a brain network as a whole graph instance into a meaningful low-dimensional representation can be used to investigate disease mechanisms and inform therapeutic interventions. Moreover, by exploiting information from multiple neuroimaging modalities or views, we are able to obtain an embedding that is more useful than the embedding learned from an individual view. Therefore, multi-view multi-graph embedding becomes a crucial task. Currently only a few studies have been devoted to this topic, and most of them focus on vector-based strategy which will cause structural information contained in the original graphs lost. As a novel attempt to tackle this problem, we propose Multi-view Multigraph Embedding (M2E) by stacking multi-graphs into multiple partially-symmetric tensors and using tensor techniques to simultaneously leverage the dependencies and correlations among multi-view and multi-graph brain networks. Extensive experiments on real HIV and bipolar disorder brain network datasets demonstrate the superior performance of M2E on clustering brain networks by leveraging the multi-view multigraph interactions. Index terms— Brain Network Embedding, Multi-graph Embedding, Tensor Factorization, Multi-view Learning

PDF Details

AAAI Conference 2016 Conference Paper

Fortune Teller: Predicting Your Career Path

Ye Liu
Luming Zhang
Liqiang Nie
Yan Yan
David Rosenblum

People go to fortune tellers in hopes of learning things about their future. A future career path is one of the topics most frequently discussed. But rather than rely on “black arts” to make predictions, in this work we scientiﬁcally and systematically study the feasibility of career path prediction from social network data. In particular, we seamlessly fuse information from multiple social networks to comprehensively describe a user and characterize progressive properties of his or her career path. This is accomplished via a multi-source learning framework with fused lasso penalty, which jointly regularizes the source and career-stage relatedness. Extensive experiments on real-world data conﬁrm the accuracy of our model.

PDF Details

AAAI Conference 2016 Conference Paper

Recognizing Complex Activities by a Probabilistic Interval-Based Model

Li Liu
Li Cheng
Ye Liu
Yongpo Jia
David Rosenblum

A key challenge in complex activity recognition is the fact that a complex activity can often be performed in several different ways, with each consisting of its own conﬁguration of atomic actions and their temporal dependencies. This leads us to deﬁne an atomic activity-based probabilistic framework that employs Allen’s interval relations to represent local temporal dependencies. The framework introduces a latent variable from the Chinese Restaurant Process to explicitly characterize these unique internal conﬁgurations of a particular complex activity as a variable number of tables. It can be analytically shown that the resulting interval network satisﬁes the transitivity property, and as a result, all local temporal dependencies can be retained and are globally consistent. Empirical evaluations on benchmark datasets suggest our approach signiﬁcantly outperforms the state-of-the-art methods.

PDF Details

TAAS Journal 2016 Journal Article

Topology Control for Diverse Coverage in Underwater Wireless Sensor Networks

Linfeng Liu
Jingli Du
Ye Liu

Underwater wireless sensor networks (UWSNs) have been developed for a set of underwater applications, including the resource exploration, pollution monitoring, tactical surveillance, and so on. However, the complexity and diversity of the underwater environment differentiate it significantly from the terrestrial environment. In particular, the coverage requirements (i.e., coverage degrees and coverage probabilities) at different regions probably differ underwater. Nevertheless, little effort has been made so far on the topology control of UWSNs given the diverse coverage requirements. To this end, this article proposes two algorithms for the diverse coverage problem in UWSNs: (1) Traversal Algorithm for Diverse Coverage (TADC), which adjusts the sensing radii of nodes successively, that is, at each round only one node alters its sensing radius, and (2) Radius Increment Algorithm for Diverse Coverage (RIADC), which sets the sensing radii of nodes incrementally, that is, at each round multiple nodes may increase their sensing radii simultaneously. The performances of TADC and RIADC are analyzed through mathematical analysis and simulations. The results reveal that both TADC and RIADC can achieve the diverse coverage while minimizing the energy consumption. Moreover, TADC and RIADC perform nicely in obtaining optimal sensing radii and reducing message complexity, respectively. Such merits further indicate that TADC and RIADC are suitable for small-scale and large-scale UWSNs, respectively.

Details DOI

IJCAI Conference 2016 Conference Paper

Urban Water Quality Prediction Based on Multi-Task Multi-View Learning

Ye Liu
Yu Zheng
Yuxuan Liang
Shuming Liu
David S. Rosenblum

Urban water quality is of great importance to our daily lives. Prediction of urban water quality help control water pollution and protect human health. In this work, we forecast the water quality of a station over the next few hours, using a multi-task multi-view learning method to fuse multiple datasets from different domains. In particular, our learning model comprises two alignments. The first alignment is the spaio-temporal view alignment, which combines local spatial and temporal information of each station. The second alignment is the prediction alignment among stations, which captures their spatial correlations and performs co-predictions by incorporating these correlations. Extensive experiments on real-world datasets demonstrate the effectiveness of our approach.

PDF Details

IJCAI Conference 2015 Conference Paper

Action2Activity: Recognizing Complex Activities from Sensor Data

Ye Liu
Liqiang Nie
Lei Han
Luming Zhang
David S. Rosenblum

As compared to simple actions, activities are much more complex, but semantically consistent with a human’s real life. Techniques for action recognition from sensor generated data are mature. However, there has been relatively little work on bridging the gap between actions and activities. To this end, this paper presents a novel approach for complex activity recognition comprising of two components. The first component is temporal pattern mining, which provides a mid-level feature representation for activities, encodes temporal relatedness among actions, and captures the intrinsic properties of activities. The second component is adaptive Multi-Task Learning, which captures relatedness among activities and selects discriminant features. Extensive experiments on a real-world dataset demonstrate the effectiveness of our work.

PDF Details