Author name cluster

Mingze Xu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

ICLR Conference 2025 Conference Paper

MM1. 5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Haotian Zhang 0005
Mingfei Gao
Zhe Gan
Philipp Dufter
Nina Wenzel
Forrest Huang
Dhruti Shah
Xianzhi Du

We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.

Details

NeurIPS Conference 2025 Conference Paper

StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant

Haibo Wang
Bo Feng
Zhengfeng Lai
Mingze Xu
Shiyu Li
Weifeng Ge
Afshin Dehghan
Meng Cao

We present StreamBridge, a simple yet effective framework that seamlessly transforms offline Video-LLMs into streaming-capable models. It addresses two fundamental challenges in adapting existing models into online scenarios: (1) limited capability for multi-turn real-time understanding, and (2) lack of proactive response mechanisms. Specifically, StreamBridge incorporates (1) a memory buffer combined with a round-decayed compression strategy, supporting long-context multi-turn interactions, and (2) a decoupled, lightweight activation model that can be effortlessly integrated into existing Video-LLMs, enabling continuous proactive responses. To further support StreamBridge, we construct Stream-IT, a large-scale dataset tailored for streaming video understanding, featuring interleaved video-text sequences and diverse instruction formats. Extensive experiments show that StreamBridge significantly improves the streaming understanding capabilities of offline Video-LLMs across various tasks, outperforming even proprietary models such as GPT-4o and Gemini 1. 5 Pro. Simultaneously, it achieves competitive or superior performance on standard video understanding benchmarks.

PDF Details

NeurIPS Conference 2025 Conference Paper

UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation

Rui Tian
Mingfei Gao
Mingze Xu
Jiaming Hu
Jiasen Lu
Zuxuan Wu
Yinfei Yang
Afshin Dehghan

We introduce UniGen, a unified multimodal large language model (MLLM) capable of image understanding and generation. We study the full training pipeline of UniGen from a data-centric perspective, including multi-stage pre-training, supervised fine-tuning, and direct preference optimization. More importantly, we propose a new Chain-of-Thought Verification (CoT-V) strategy for test-time scaling, which significantly boosts UniGen’s image generation quality using a simple Best-of-N test-time strategy. Specifically, CoT-V enables UniGen to act as both image generator and verifier at test time, assessing the semantic alignment between a text prompt and its generated image in a step-by-step CoT manner. Trained entirely on open-source datasets across all stages, UniGen achieves state-of-the-art performance on a range of image understanding and generation benchmarks, with a final score of 0. 78 on GenEval and 85. 19 on DPG-Bench. Through extensive ablation studies, our work provides actionable insights and addresses key challenges in the full life cycle of building unified MLLMs, contributing meaningful directions to future research. Code is available at https: //github. com/apple/ml-unigen.

PDF Details

EAAI Journal 2024 Journal Article

Resilient dynamic microgrid formation by deep reinforcement learning integrating physics-informed neural networks

Mingze Xu
Shunbo Lei
Chong Wang
Liang Liang
Junhua Zhao
Chaoyi Peng

Dynamic microgrid formation can enhance topological flexibility within the distribution system, particularly during extreme events, thereby facilitating a more efficient restoration process. However, existing research has overlooked the impact of cold load pickup on system restoration efforts. A sudden load spike can lead to the overloading of generators and transformers, which can result in the failure of the system restoration process. This study leverages the topological flexibility through dynamic microgrid formation of the system to mitigate the impact of cold load pickup, thereby enhancing the efficiency of sequential load restoration. To alleviate the computational complexity arising from intricate operational constraints and the uncertainties inherent in cold load pickup conditions, this paper proposes a novel model-free framework. Unlike existing deep reinforcement learning models, we incorporate physical constraint information into the model by means of physics-informed neural networks, where the solution of an optimization problem is regarded as knowledge, enabling the agent to learn operational constraints more efficiently and stably. The proposed approach is compatible with and can be integrated into any deep reinforcement learning algorithm that utilizes the advantage actor–critic framework with neural networks. This research employs the deep deterministic policy gradient algorithm as a representative example for investigation. The effectiveness and generalization performance of the proposed method are validated on a modified IEEE 123-node test feeder, while its scalability is assessed using the IEEE 8500-node test feeder system.

Details DOI

NeurIPS Conference 2022 Conference Paper

An In-depth Study of Stochastic Backpropagation

Jun Fang
Mingze Xu
Hao Chen
Bing Shuai
Zhuowen Tu
Joseph Tighe

In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates gradients by using only a subset of feature maps to save GPU memory and computational cost. We interpret SBP as an efficient way to implement stochastic gradient decent by performing backpropagation dropout, which leads to significant memory saving and training run-time reduction, with a minimal impact on the overall model accuracy. We offer best practices to apply SBP for training image recognition models, which can be adopted in learning a wide range of deep neural networks. Experiments on image classification and object detection show that SBP can save up to 40% of GPU memory with less than 1% accuracy degradation. Code is available at: https: //github. com/amazon-research/stochastic-backpropagation

PDF Details

NeurIPS Conference 2021 Conference Paper

Long Short-Term Transformer for Online Action Detection

Mingze Xu
Yuanjun Xiong
Hao Chen
Xinyu Li
Wei Xia
Zhuowen Tu
Stefano Soatto

We present Long Short-term TRansformer (LSTR), a temporal modeling algorithm for online action detection, which employs a long- and short-term memory mechanism to model prolonged sequence data. It consists of an LSTR encoder that dynamically leverages coarse-scale historical information from an extended temporal window (e. g. , 2048 frames spanning of up to 8 minutes), together with an LSTR decoder that focuses on a short time window (e. g. , 32 frames spanning 8 seconds) to model the fine-scale characteristics of the data. Compared to prior work, LSTR provides an effective and efficient method to model long videos with fewer heuristics, which is validated by extensive empirical analysis. LSTR achieves state-of-the-art performance on three standard online action detection benchmarks, THUMOS'14, TVSeries, and HACS Segment. Code has been made available at: https: //xumingze0308. github. io/projects/lstr.

PDF Details

ICRA Conference 2019 Conference Paper

Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems

Yu Yao 0006
Mingze Xu
Chiho Choi
David J. Crandall
Ella M. Atkins
Behzad Dariush

Predicting the future location of vehicles is essential for safety-critical applications such as advanced driver assistance systems (ADAS) and autonomous driving. This paper introduces a novel approach to simultaneously predict both the location and scale of target vehicles in the first-person (egocentric) view of an ego-vehicle. We present a multi-stream recurrent neural network (RNN) encoder-decoder model that separately captures both object location and scale and pixel-level observations for future vehicle localization. We show that incorporating dense optical flow improves prediction results significantly since it captures information about motion as well as appearance change. We also find that explicitly modeling future motion of the ego-vehicle improves the prediction accuracy, which could be especially beneficial in intelligent and automated vehicles that have motion planning capability. To evaluate the performance of our approach, we present a new dataset of first-person videos collected from a variety of scenarios at road intersections, which are particularly challenging moments for prediction because vehicle trajectories are diverse and dynamic. Code and dataset have been made available at: https://usa.honda-ri.com/hevi.

Details

IROS Conference 2019 Conference Paper

Unsupervised Traffic Accident Detection in First-Person Videos

Yu Yao 0006
Mingze Xu
Yuchen Wang
David J. Crandall
Ella M. Atkins

Recognizing abnormal events such as traffic violations and accidents in natural driving scenes is essential for successful autonomous driving and advanced driver assistance systems. However, most work on video anomaly detection suffers from two crucial drawbacks. First, they assume cameras are fixed and videos have static backgrounds, which is reasonable for surveillance applications but not for vehicle-mounted cameras. Second, they pose the problem as one-class classification, relying on arduously hand-labeled training datasets that limit recognition to anomaly categories that have been explicitly trained. This paper proposes an unsupervised approach for traffic accident detection in first-person (dashboard-mounted camera) videos. Our major novelty is to detect anomalies by predicting the future locations of traffic participants and then monitoring the prediction accuracy and consistency metrics with three different strategies. We evaluate our approach using a new dataset of diverse traffic accidents, AnAn Accident Detection (A3D), as well as another publicly-available dataset. Experimental results show that our approach outperforms the state-of-the-art. Code and the dataset developed in this work are available at: https://github.com/MoonBlvd/tad-IROS2019

Details