Author name cluster

Dongxu Li

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

13 papers

2 author rows

JBHI Journal 2026 Journal Article

LLM-DDI: Leveraging Large Language Models for Drug-Drug Interaction Prediction on Biomedical Knowledge Graph

Dongxu Li
Yue Yang
Ziwen Cui
Hengchuang Yin
Pengwei Hu
Lun Hu

Drug-drug interaction (DDI) refers to the interaction relationships between drugs. Discovering new DDIs is crucial for advancing drug development and enhancing clinical treatments. Given the significant progress achieved through graph neural networks (GNNs), network-based models have become a prevalent approach for tackling this challenge. However, current network-based approaches are incapable of seamlessly integrating a wide range of information. Motivated by this discovery, we propose a novel model, namely LLM-DDI, which aims to comprehensively tackle DDI prediction tasks by integrating various information of molecules in the BKG. LLM-DDI initially incorporates the generative pre-trained transformer (GPT) model to generate embeddings for each molecule within the biomedical knowledge graph (BKG). These embeddings encompass diverse types of information pertaining to each molecule. Subsequently, LLM-DDI utilizes a message-passing GNN framework to enhance the learning of molecular representations with the embeddings derived from GPT as input. LLM-DDI governs the propagation of information within the BKG by semantic relationships. These semantic relationships determine how information flows and is exchanged between different entities in the BKG. Finally, LLM-DDI leverages the learned drug representations to predict potential DDIs. Experiments show the effectiveness of LLM-DDI, as it achieves the best performance on two real-world datasets, providing valuable guidance for drug development and clinical treatment.

Details DOI

JBHI Journal 2026 Journal Article

Multi-View Contrastive Learning for Drug-Drug Interaction Event Prediction

Dongxu Li
Feifan Zhao
Yue Yang
Ziwen Cui
Pengwei Hu
Lun Hu

Drug-drug interactions (DDIs) represent a critical challenge in pharmacology, often leading to adverse effects and compromised therapeutic efficacy. Accurate prediction of DDI events, which involve not only identifying interacting drug pairs but also characterizing the specific nature and context of their interactions, is essential for drug safety and personalized medicine. In this study, we propose a novel Multi-view Contrastive Learning framework, namely MCL-DDI, for DDI Event Prediction by leveraging multi-view representations of drugs to enhance predictive performance. MCL-DDI integrates molecular structures and network features, capturing complementary information about drug properties and interactions. By employing contrastive learning, we align and unify drug representations across these diverse views, enabling the framework to distinguish complex interaction patterns. Extensive experiments on benchmark datasets demonstrate that MCL-DDI outperforms state-of-the-art methods in terms of predictive accuracy. Furthermore, case studies highlight the model’s ability to identify clinically relevant DDIs, offering practical insights for drug development and risk assessment. Our work establishes a robust and accurate paradigm for DDI event prediction, paving the way for safer and more effective pharmacological interventions.

Details DOI

NeurIPS Conference 2024 Conference Paper

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

Haoning Wu
Dongxu Li
Bei Chen
Junnan Li

Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate this gap, we introduce LongVideoBench, a question-answering benchmark that features video-language interleaved inputs up to an hour long. Our benchmark includes 3, 763 varying-length web-collected videos with their subtitles across diverse themes, designed to comprehensively evaluate LMMs on long-term multimodal understanding. To achieve this, we interpret the primary challenge as to accurately retrieve and reason over detailed multimodal information from long inputs. As such, we formulate a novel video question-answering task termed referring reasoning. Specifically, as part of the question, it contains a referring query that references related video contexts, called referred context. The model is then required to reason over relevant video details from the referred context. Following the paradigm of referring reasoning, we curate 6, 678 human-annotated multiple-choice questions in 17 fine-grained categories, establishing one of the most comprehensive benchmarks for long-form video understanding. Evaluations suggest that the LongVideoBench presents significant challenges even for the most advanced proprietary models (e. g. GPT-4o, Gemini-1. 5-Pro), while their open-source counterparts show an even larger performance gap. In addition, our results indicate that model performance on the benchmark improves only when they are capable of processing more frames, positioning LongVideoBench as a valuable benchmark for evaluating future-generation long-context LMMs.

PDF Details DOI

YNIMG Journal 2024 Journal Article

Wireless optically pumped magnetometer MEG

Hao Cheng
Kaiyan He
Congcong Li
Xiao Ma
Fufu Zheng
Wei Xu
Pan Liao
Rui Yang

The current magnetoencephalography (MEG) systems, which rely on cables for control and signal transmission, do not fully realize the potential of wearable optically pumped magnetometers (OPM). This study presents a significant advancement in wireless OPM-MEG by reducing magnetization in the electronics and developing a tailored wireless communication protocol. Our protocol effectively eliminates electromagnetic interference, particularly in the critical frequency bands of MEG signals, and accurately synchronizes the acquisition and stimulation channels with the host computer's clock. We have successfully achieved single-channel wireless OPM-MEG measurement and demonstrated its reliability by replicating three well-established experiments: The alpha rhythm, auditory evoked field, and steady-state visual evoked field in the human brain. Our prototype wireless OPM-MEG system not only streamlines the measurement process but also represents a major step forward in the development of wearable OPM-MEG applications in both neuroscience and clinical research.

Details DOI

NeurIPS Conference 2023 Conference Paper

BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing

Dongxu Li
Junnan Li
Steven Hoi

Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Implementations are available at: https: //github. com/salesforce/LAVIS/tree/main/projects/blip-diffusion.

PDF Details

NeurIPS Conference 2023 Conference Paper

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Wenliang Dai
Junnan Li
Dongxu Li
Anthony Tiong
Junqi Zhao
Weisheng Wang
Boyang Li
Pascale N Fung

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e. g. , 90. 7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-source.

PDF Details

TMLR Journal 2023 Journal Article

Linearized Relative Positional Encoding

Zhen Qin
Weixuan Sun
Kaiyue Lu
Hui Deng
Dongxu Li
Xiaodong Han
Yuchao Dai
Lingpeng Kong

Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are not always directly applicable to a linear transformer, because the latter requires a decomposition of the query and key representations into separate kernel functions. Nevertheless, principles for designing encoding methods suitable for linear transformers remain understudied. In this work, we put together a variety of existing linear relative positional encoding approaches under a canonical form and further propose a family of linear relative positional encoding algorithms via unitary transformation. Our formulation leads to a principled framework that can be used to develop new relative positional encoding methods that preserve linear space-time complexity. Equipped with different models, the proposed linearized relative positional encoding (LRPE) family derives effective encoding for various applications. Experiments show that compared with existing methods, LRPE achieves state-of-the-art performance in language modeling, text classification, and image classification. Meanwhile, it emphasizes a general paradigm for designing broadly more relative positional encoding methods that are applicable to linear transformers.

PDF Details

ICRA Conference 2023 Conference Paper

TODE-Trans: Transparent Object Depth Estimation with Transformer

Kang Chen
Shaochen Wang
Beihao Xia
Dongxu Li
Zhen Kan
Bin Li 0025

Transparent objects are widely used in industrial automation and daily life. However, robust visual recognition and perception of transparent objects have always been a major challenge. Currently, most commercial-grade depth cameras are still not good at sensing the surfaces of transparent objects due to the refraction and reflection of light. In this work, we present a transformer-based transparent object depth estimation approach from a single RGB-D input. We observe that the global characteristics of the transformer make it easier to extract contextual information to perform depth estimation of transparent areas. In addition, to better enhance the fine-grained features, a feature fusion module (FFM) is designed to assist coherent prediction. Our empirical evidence demonstrates that our model delivers significant improvements in recent popular datasets, e. g. , 25% gain on RMSE and 21% gain on REL compared to previous state-of-the-art convolutional-based counterparts in ClearGrasp dataset. Extensive results show that our transformer-based model enables better aggregation of the object's RGB and inaccurate depth information to obtain a better depth representation. Our code and the pre-trained model are available at https://github.com/yuchendoudou/TODE.

Details

YNIMG Journal 2022 Journal Article

Multimodal neuroimaging with optically pumped magnetometers: A simultaneous MEG-EEG-fNIRS acquisition system

Xingyu Ru
Kaiyan He
Bingjiang Lyu
Dongxu Li
Wei Xu
Wenyu Gu
Xiao Ma
Jiayi Liu

Multimodal neuroimaging plays an important role in neuroscience research. Integrated noninvasive neuroimaging modalities, such as magnetoencephalography (MEG), electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), allow neural activity and related physiological processes in the brain to be precisely and comprehensively depicted, providing an effective and advanced platform to study brain function. Noncryogenic optically pumped magnetometer (OPM) MEG has high signal power due to its on-scalp sensor layout and enables more flexible configurations than traditional commercial superconducting MEG. Here, we integrate OPM-MEG with EEG and fNIRS to develop a multimodal neuroimaging system that can simultaneously measure brain electrophysiology and hemodynamics. We conducted a series of experiments to demonstrate the feasibility and robustness of our MEG-EEG-fNIRS acquisition system. The complementary neural and physiological signals simultaneously collected by our multimodal imaging system provide opportunities for a wide range of potential applications in neurovascular coupling, wearable neuroimaging, hyperscanning and brain-computer interfaces.

Details DOI

AAAI Conference 2022 Conference Paper

Towards Explainable Action Recognition by Salient Qualitative Spatial Object Relation Chains

Hua Hua
Dongxu Li
Ruiqi Li
Peng Zhang
Jochen Renz
Anthony Cohn

In order to be trusted by humans, Artificial Intelligence agents should be able to describe rationales behind their decisions. One such application is human action recognition in critical or sensitive scenarios, where trustworthy and explainable action recognizers are expected. For example, reliable pedestrian action recognition is essential for self-driving cars and explanations for real-time decision making are critical for investigations if an accident happens. In this regard, learningbased approaches, despite their popularity and accuracy, are disadvantageous due to their limited interpretability. This paper presents a novel neuro-symbolic approach that recognizes actions from videos with human-understandable explanations. Specifically, we first propose to represent videos symbolically by qualitative spatial relations between objects called qualitative spatial object relation chains. We further develop a neural saliency estimator to capture the correlation between such object relation chains and the occurrence of actions. Given an unseen video, this neural saliency estimator is able to tell which object relation chains are more important for the action recognized. We evaluate our approach on two real-life video datasets, with respect to recognition accuracy and the quality of generated action explanations. Experiments show that our approach achieves superior performance on both aspects to previous symbolic approaches, thus facilitating trustworthy intelligent decision making. Our approach can be used to augment state-of-the-art learning approaches with explainability.

PDF Details

AAAI Conference 2022 Conference Paper

Transcribing Natural Languages for the Deaf via Neural Editing Programs

Dongxu Li
Chenchen Xu
Liu Liu
Yiran Zhong
Rong Wang
Lars Petersson
Hongdong Li

This work studies the task of glossification, of which the aim is to transcribe natural spoken language sentences for the Deaf (hard-of-hearing) community to ordered sign language glosses. Previous sequence-to-sequence language models trained with paired sentence-gloss data often fail to capture the rich connections between the two distinct languages, leading to unsatisfactory transcriptions. We observe that despite different grammars, glosses effectively simplify sentences for the ease of deaf communication, while sharing a large portion of vocabulary with sentences. This has motivated us to implement glossification by executing a collection of editing actions, e. g. word addition, deletion and copying, called editing programs, on their natural spoken language counterparts. Specifically, we design a new neural agent that learns to synthesize and execute editing programs, conditioned on sentence contexts and partial editing results. The agent is trained to imitate minimal editing programs, while exploring more widely the program space via policy gradients to optimize sequence-wise transcription quality. Results show that our approach outperforms previous glossification models by a large margin, improving the BLEU-4 score from 16. 45 to 18. 89 on RWTH-PHOENIX- WEATHER-2014T and from 18. 38 to 21. 30 on CSL-Daily.

PDF Details

NeurIPS Conference 2020 Conference Paper

TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation

Dongxu Li
Chenchen Xu
Xin Yu
Kaihao Zhang
Benjamin Swift
Hanna Suominen
Hongdong Li

Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with no clear boundaries in between. Existing SLT models usually represent sign visual features in a frame-wise manner so as to avoid needing to explicitly segmenting the videos into isolated signs. However, these methods neglect the temporal information of signs and lead to substantial ambiguity in translation. In this paper, we explore the temporal semantic structures of sign videos to learn more discriminative features. To this end, we first present a novel sign video segment representation which takes into account multiple temporal granularities, thus alleviating the need for accurate video segmentation. Taking advantage of the proposed segment representation, we develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet. Specifically, TSPNet introduces an inter-scale attention to evaluate and enhance local semantic consistency of sign segments and an intra-scale attention to resolve semantic ambiguity by using non-local video context. Experiments show that our TSPNet outperforms the state-of-the-art with significant improvements on the BLEU score (from 9. 58 to 13. 41) and ROUGE score (from 31. 80 to 34. 96) on the largest commonly used SLT dataset. Our implementation is available at https: //github. com/verashira/TSPNet.

PDF Details

IJCAI Conference 2018 Conference Paper

Effect-Abstraction Based Relaxation for Linear Numeric Planning

Dongxu Li
Enrico Scala
Patrik Haslum
Sergiy Bogomolov

This paper studies an effect-abstraction based relaxation for reasoning about linear numeric planning problems. The effect-abstraction decomposes non-constant linear numeric effects into actions with conditional effects over additive constant numeric effects. With little effort, on this compiled version, it is possible to use known subgoaling based relaxations and relative heuristics. The combination of these two steps leads to a novel relaxation based heuristic. Theoretically, the relaxation is proved tighter than previous interval based relaxation and leading to safe-pruning heuristics. Empirically, a heuristic developed on this relaxation leads to substantial improvements for a class of problems that are currently out of the reach of state-of-the-art numeric planners.

PDF Details