Author name cluster

Yiwei Lyu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

13 papers

1 author row

TMLR Journal 2026 Journal Article

Towards Scalable Language-Image Pre-training for 3D Medical Imaging

Chenhui Zhao
Yiwei Lyu
Asadur Zaman Chowdury
Edward S Harake
Akhil Kondepudi
Akshay T Rao
Xinhai Hou
Honglak Lee

The scalability of current language-image pre-training for 3D medical imaging, such as CT and MRI, is constrained by the need for radiologists to manually curate raw clinical studies. In this work, we pioneer pre-training directly on uncurated studies, which both aligns more closely with the clinical workflow and provides a natural path to scalability. However, the unique structure of such data presents new challenges for existing model architectures, which were originally designed for 2D slices or single 3D scans. To address this, we introduce a novel hierarchical attention mechanism inspired by the intrinsic hierarchy of radiology data: slice, scan, and study. We denote our framework as Hierarchical attention for Language-Image Pre-training (HLIP). Trained on 220K studies with 3.13 million scans for brain MRI and 240K studies with 1.44 million scans for head CT, HLIP achieves state-of-the-art performance, e.g., +10.5% balanced ACC on the proposed publicly available brain MRI benchmark Pub-Brain-5; +8.3% and +1.7% macro AUC on head CT benchmarks CQ500 and RSNA, respectively. HLIP also exhibits strong generalizability on existing 3D medical language-image pre-training benchmarks, e.g., +4.3% macro AUC on the Rad-ChestCT benchmark when pre-trained on CT-RATE. These results demonstrate that, with HLIP, directly pre-training on uncurated clinical datasets is a scalable and effective direction for language-image pre-training in 3D medical imaging. The code is available at https://github.com/zch0414/hlip

PDF Details

AAAI Conference 2025 Conference Paper

Step-Calibrated Diffusion for Biomedical Optical Image Restoration

Yiwei Lyu
Sung Jik Cha
Cheng Jiang
Asadur Zaman Chowdury
Xinhai Hou
Edward S. Harake
Akhil Kondepudi
Christian Freudiger

High-quality, high-resolution medical imaging is essential for clinical care. Raman-based biomedical optical imaging uses non-ionizing infrared radiation to evaluate human tissues in real time and is used for early cancer detection, brain tumor diagnosis, and intraoperative tissue analysis. Unfortunately, optical imaging is vulnerable to image degradation due to laser scattering and absorption, which can result in diagnostic errors and misguided treatment. Restoration of optical images is a challenging computer vision task because the sources of image degradation are multi-factorial, stochastic, and tissue-dependent, preventing a straightforward method to obtain paired low-quality/high-quality data. Here, we present Restorative Step-Calibrated Diffusion (RSCD): an unpaired diffusion-based image restoration method that uses a step calibrator model to dynamically determine the number of steps required to complete the reverse diffusion process for image restoration. RSCD outperforms other widely used unpaired image restoration methods on both image quality and perceptual evaluation metrics for restoring optical images. Medical imaging experts consistently prefer images restored using RSCD in blinded comparison experiments and report minimal to no hallucinations. Finally, we show that RSCD improves performance on downstream clinical imaging tasks, including automated brain tumor diagnosis and deep tissue imaging.

PDF Details DOI

AAMAS Conference 2024 Conference Paper

Interactive Control and Decision-Making for Multi-Robots Systems

Yiwei Lyu

In order to achieve the ultimate goal of harmonious human-robot co-existence, the key is to build autonomous robots that can safely interact with humans for collaboration and coordination, as well as demonstrate reliable behavior that is acceptable to humans. These two requirements slightly differ from each other, with the former addressing the safety and functionality of robots as task performers, and the latter emphasizing the social compliance of robots as entities in society. In this abstract, I will outline my efforts towards enhancing the safety and reliability of interactive robot autonomy from three progressively advancing perspectives, 1) self-level autonomy, aiming to develop reactive behavior that ensures safety for individual robots when encountering non-cooperative agents, 2) peer-level autonomy, emphasizing the establishment of a safe interaction mechanism within an diverse and unconnected multirobot system, and 3) human-involved autonomy, highlighting the consideration of human factors in the decision-making process for the design of multi-robot systems.

PDF

AAMAS Conference 2023 Conference Paper

Decentralized Safe Navigation for Multi-agent Systems via Risk-aware Weighted Buffered Voronoi Cells

Yiwei Lyu
John M. Dolan
Wenhao Luo

In this paper, we propose Risk-aware Weighted Buffered Voronoi tessellation, a variant of Generalized Voronoi tessellation, for decentralized multi-agent collision-free navigation. Inherited from the traditional Voronoi tessellation, a safety guarantee in terms of inter-robot collision avoidance is achieved by partitioning the joint state space of the multi-agent system into individual cells that constrain each individual agent’s motion in a distributed manner. Different from many existing Voronoi tessellations-based collision avoidance approaches, our Risk-aware Weighted Buffered Voronoi Cell (Risk-aware WBVC) partition not only takes agent positional information into account, but also the motion information when determining the cell boundaries between pairwise robots. Our riskaware WBVC relies on the novel use of Control Barrier Functions (CBF) as a measure of risk evaluation that captures to what extent the safety constraints are satisfied between pairwise robots. With that, the cell boundaries of risk-aware WBVC are determined by (1) the varying levels of relative efforts between pairwise agents to respond to potential collisions, and (2) the accumulated risk each agent experiences that is caused by the surrounding agents. This allows for an adaptive constrained space partition among robots that balances between individual’s efforts in respecting the safety constraints and the overall threats due to other agents in the environment, e. g. an aggressive robot moving with higher speed requires a relatively larger space for responding to potential collisions, and a less-threatened robot may be expected to yield and make more room for those exposed to higher risk. Rigorous proofs of formal safety guarantees are provided and simulations are demonstrated on up to 16 robots to show the effectiveness of our method.

PDF

TMLR Journal 2023 Journal Article

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Paul Pu Liang
Yiwei Lyu
Xiang Fan
Jeffrey Tsaw
Yudong Liu
Shentong Mo
Dani Yogatama
Louis-Philippe Morency

Many real-world problems are inherently multimodal, from the communicative modalities humans use to express social and emotional states such as spoken language, gestures, and paralinguistics to the force, proprioception, and visual sensors ubiquitous on robots. While there has been an explosion of interest in multimodal representation learning, these methods are still largely focused on a small set of modalities, primarily in the language, vision, and audio space. In order to accelerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. Since adding new models for every new modality or task becomes prohibitively expensive, a critical technical challenge is heterogeneity quantification: how can we measure which modalities encode similar information and interactions in order to permit parameter sharing with previous modalities? This paper proposes two new information theoretic metrics for heterogeneity quantification: (1) modality heterogeneity studies how similar $2$ modalities $\{X_1,X_2\}$ are by measuring how much information can be transferred from $X_1$ to $X_2$, while (2) interaction heterogeneity studies how similarly pairs of modalities $\{X_1,X_2\}, \{X_3,X_4\}$ interact by measuring how much interaction information can be transferred from $\{X_1,X_2\}$ to $\{X_3,X_4\}$. We show the importance of these $2$ proposed metrics in high-modality scenarios as a way to automatically prioritize the fusion of modalities that contain unique information or unique interactions. The result is a single model, HighMMT, that scales up to $10$ modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and $15$ tasks from $5$ different research areas. Not only does HighMMT outperform prior methods on the tradeoff between performance and efficiency, it also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning. We release our code and benchmarks, which we hope will present a unified platform for subsequent theoretical and empirical analysis.

PDF Details

AAMAS Conference 2023 Conference Paper

Minimally Constraining Line-of-Sight Connectivity Maintenance for Collision-free Multi-Robot Networks under Uncertainty

Yupeng Yang
Yiwei Lyu
Wenhao Luo

In this paper, we consider the Line-of-Sight (LOS) connectivity maintenance under positional uncertainty for a team of robots consisting of multiple subgroups with given parallel tasks. The LOS connectivity between pairwise robots is preserved when the two robots are within the limited communication range and their LOS is occlusion-free from static obstacles over time. By unifying a control theoretic approach and a graph theoretic approach, we develop an Uncertainty Aware Line-of-Sight Minimum Spanning Tree (LOS- MST) framework to compute robots’ motion that maintains only a minimally constraining set of LOS edges among robots for global and subgroup LOS connectivity, while minimizing the motion disruption to their original multi-robot behaviors. Simulation results are provided to validate the effectiveness of our proposed approach.

PDF

AAMAS Conference 2023 Conference Paper

Model-based Dynamic Shielding for Safe and Efficient Multi-agent Reinforcement Learning

Wenli Xiao
Yiwei Lyu
John Dolan

Multi-Agent Reinforcement Learning (MARL) discovers policies that maximize reward but do not have safety guarantees during the learning and deployment phases. Although shielding with Linear Temporal Logic (LTL) is a promising formal method to ensure safety in single-agent Reinforcement Learning (RL), it results in conservative behaviors when scaling to multi-agent scenarios. Additionally, it poses computational challenges for synthesizing shields in complex multi-agent environments. This work introduces Model-based Dynamic Shielding (MBDS) to support MARL algorithm design. Our algorithm synthesizes distributive shields, which are reactive systems running in parallel with each MARL agent, to monitor and rectify unsafe behaviors. The shields can dynamically split, merge, and recompute based on agents’ states. This design enables efficient synthesis of shields to monitor agents in complex environments without coordination overheads. We also propose an algorithm to synthesize shields without prior knowledge of the dynamics model. The proposed algorithm obtains an approximate world model by interacting with the environment during the early stage of exploration, making our MBDS enjoy formal safety guarantees with high probability. We demonstrate in simulations that our framework can surpass existing baselines in terms of safety guarantees and learning performance.

PDF

JMLR Journal 2023 Journal Article

MultiZoo and MultiBench: A Standardized Toolkit for Multimodal Deep Learning

Paul Pu Liang
Yiwei Lyu
Xiang Fan
Arav Agarwal
Yun Cheng
Louis-Philippe Morency
Ruslan Salakhutdinov

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiZoo, a public toolkit consisting of standardized implementations of >20 core multimodal algorithms and MultiBench, a large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. Together, these provide an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, we offer a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench paves the way towards a better understanding of the capabilities and limitations of multimodal models, while ensuring ease of use, accessibility, and reproducibility. Our toolkits are publicly available, will be regularly updated, and welcome inputs from the community. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2023. ( edit, beta )

PDF Details

AAAI Conference 2023 Short Paper

Risk-Aware Decentralized Safe Control via Dynamic Responsibility Allocation (Student Abstract)

Yiwei Lyu
Wenhao Luo
John M. Dolan

In this work, we present a novel risk-aware decentralized Control Barrier Function (CBF)-based controller for multi-agent systems. The proposed decentralized controller is composed based on pairwise agent responsibility shares (a percentage), calculated from the risk evaluation of each individual agent faces in a multi-agent interaction environment. With our proposed CBF-inspired risk evaluation framework, the responsibility portions between pairwise agents are dynamically updated based on the relative risk they face. Our method allows agents with lower risk to enjoy a higher level of freedom in terms of a wider action space, and the agents exposed to higher risk are constrained more tightly on action spaces, and are therefore forced to proceed with caution.

PDF Details DOI

AAAI Conference 2023 Short Paper

Safe Interactive Autonomy for Multi-Agent Systems

Yiwei Lyu

It is envisioned that in the near future autonomous systems such as multi-agent systems, will co-exist with humans, e.g., autonomous vehicles will share roads with human drivers. These safety-critical scenarios require formally provable safety guarantees so that the robots will never collide with humans or with each other. It is challenging to provide such guarantees in the real world due to the stochastic environments and inaccurate models of heterogeneous agents including robots and humans. My PhD research investigates decision-making algorithm design for provably-correct safety guarantees in mixed multi-agent systems.

PDF Details DOI

AAAI Conference 2023 Short Paper

Tackling Safe and Efficient Multi-Agent Reinforcement Learning via Dynamic Shielding (Student Abstract)

Wenli Xiao
Yiwei Lyu
John M. Dolan

Multi-agent Reinforcement Learning (MARL) has been increasingly used in safety-critical applications but has no safety guarantees, especially during training. In this paper, we propose dynamic shielding, a novel decentralized MARL framework to ensure safety in both training and deployment phases. Our framework leverages Shield, a reactive system running in parallel with the reinforcement learning algorithm to monitor and correct agents' behavior. In our algorithm, shields dynamically split and merge according to the environment state in order to maintain decentralization and avoid conservative behaviors while enjoying formal safety guarantees. We demonstrate the effectiveness of MARL with dynamic shielding in the mobile navigation scenario.

PDF Details DOI

AAAI Conference 2022 Short Paper

Adaptive Safe Behavior Generation for Heterogeneous Autonomous Vehicles Using Parametric-Control Barrier Functions (Student Abstract)

Yiwei Lyu
Wenhao Luo
John M. Dolan

Control Barrier Functions have been extensively studied to ensure guaranteed safety during inter-robot interactions. In this paper, we introduce the Parametric-Control Barrier Function (Parametric-CBF), a novel variant of the traditional Control Barrier Function to extend its expressivity in describing different safe behaviors among heterogeneous robots. A parametric-CBF based framework is presented to enable the ego robot to model the neighboring robots behavior and further improve the coordination efficiency during interaction while enjoying formally provable safety guarantees. We demonstrate the usage of Parametric-CBF in behavior prediction and adaptive safe control in the ramp merging scenario.

PDF Details

NeurIPS Conference 2021 Conference Paper

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Paul Pu Liang
Yiwei Lyu
Xiang Fan
Zetian Wu
Yun Cheng
Jason Wu
Leslie (Yufan) Chen
Peter Wu

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark for multimodal learning spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning spanning innovations in fusion paradigms, optimization objectives, and training approaches. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal machine learning research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized implementations, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

PDF Details