Author name cluster

Jie Ji

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

2 author rows

TMLR Journal 2026 Journal Article

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning

Mingyu Cao
Gen Li
Jie Ji
Jiaqi Zhang
AJAY JAISWAL
Li Shen
Xiaolong Ma
Shiwei Liu

Mixture-of-Experts (MoE) has garnered significant attention for its ability to scale up neural networks while utilizing the same or even fewer active parameters. However, MoE does not alleviate the massive memory requirements of networks, which limits their practicality in real-world applications, especially in the era of large language models (LLMs). While recent work explores the possibility of removing entire layers of MoE to reduce memory, the performance degradation is still notable. In this paper, we propose ConDense-MoE (CD-MoE), which, instead of dropping the entire MoE layer, condenses the large, sparse MoE layer into a smaller, denser layer with only a few experts activated for all tokens, while maintaining hardware friendliness. Our approach is specifically designed for fine-grained MoE with shared experts, where Feed-Forward Networks are split into many small experts, with certain experts isolated to serve as shared experts that are always activated, such as DeepSeekMoE and QwenMoE. We demonstrate the effectiveness of our method. Specifically, for the DeepSeekMoE-16B model, our approach maintains 90% of the average accuracy while reducing memory usage by 27.5% and increasing inference speed by 1.26 times. Moreover, we show that by applying lightweight expert fine-tuning—only to the condensed layers—and using 5 hours on a single 80G A100 GPU, we can successfully recover 98% of the original performance.

PDF Details

EAAI Journal 2025 Journal Article

Applications of machine vision technology for conveyor belt deviation detection: A review and roadmap

Jiaming Han
Ting Fang
Wensheng Liu
Chenxiao Zhang
Molin Zhu
Jibin Xu
Jie Ji
Xianhua He

Conveyor belt deviation is a frequent challenge in product transportation filed, and failure to promptly detect and rectify this anomaly not only significantly reduces transport efficiency but also poses a risk of serious safety accidents, leading to enormous economic losses. Traditional contact-based deviation detection technologies, with their inherent limitations of high costs and complicated maintenance, have struggled to meet the practical demands of long-distance conveyor belt inspection. In this context, non-contact machine vision technology has emerged as a prominent solution in the field of conveyor belt deviation detection, thanks to its notable advantages of a simple hardware structure and round-the-clock operational capability. Recently, with the rapid development of artificial intelligence theories, this research field has accumulated a series of effective solutions that have been proven through machine vision practical applications. This paper delves into the technical principles of the existing solutions, systematically summarizes them and objectively evaluates their strengths and weaknesses in practical applications. Based on this foundation, this paper also provides an insight on the future development trends of intelligent monitoring for conveyor belt deviation, aiming to offer valuable reference and guidance to technicians in related fields.

Details DOI

NeurIPS Conference 2025 Conference Paper

Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning

Hossein Rajoli Nowdeh
Jie Ji
Xiaolong Ma
Fatemeh Afghah

In multimodal learning, dominant modalities often overshadow others, limiting generalization. We propose Modality-Aware Sharpness-Aware Minimization (M-SAM), a model-agnostic framework that applies to many modalities and supports early and late fusion scenarios. In every iteration, M-SAM in three steps optimizes learning. \textbf{First, it identifies the dominant modality} based on modalities' contribution in the accuracy using Shapley. \textbf{Second, it decomposes the loss landscape}, or in another language, it modulates the loss to prioritize the robustness of the model in favor of the dominant modality, and \textbf{third, M-SAM updates the weights} by backpropagation of modulated gradients. This ensures robust learning for the dominant modality while enhancing contributions from others, allowing the model to explore and exploit complementary features that strengthen overall performance. Extensive experiments on four diverse datasets show that M-SAM outperforms the latest state-of-the-art optimization and gradient manipulation methods and significantly balances and improves multimodal learning. The code will be released.

PDF Details

NeurIPS Conference 2024 Conference Paper

A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training

Jie Ji
Gen Li
Jingjing Fu
Fatemeh Afghah
Linke Guo
Xiaoyong Yuan
Xiaolong Ma

Sparse training stands as a landmark approach in addressing the considerable training resource demands imposed by the continuously expanding size of Deep Neural Networks (DNNs). However, the training of a sparse DNN encounters great challenges in achieving optimal generalization ability despite the efforts from the state-of-the-art sparse training methodologies. To unravel the mysterious reason behind the difficulty of sparse training, we connect the network sparsity with neural loss functions structure, and identify the cause of such difficulty lies in chaotic loss surface. In light of such revelation, we propose $S^{2} - SAM$, characterized by a **S**ingle-step **S**harpness_**A**ware **M**inimization that is tailored for **S**parse training. For the first time, $S^{2} - SAM$ innovates the traditional SAM-style optimization by approximating sharpness perturbation through prior gradient information, incurring *zero extra cost*. Therefore, $S^{2} - SAM$ not only exhibits the capacity to improve generalization but also aligns with the efficiency goal of sparse training. Additionally, we study the generalization result of $S^{2} - SAM$ and provide theoretical proof for convergence. Through extensive experiments, $S^{2} - SAM$ demonstrates its universally applicable plug-and-play functionality, enhancing accuracy across various sparse training methods. Code available at https: //github. com/jjsrf/SSAM-NEURIPS2024.

PDF Details DOI

ICML Conference 2024 Conference Paper

Advancing Dynamic Sparse Training by Exploring Optimization Opportunities

Jie Ji
Gen Li 0012
Lu Yin 0006
Minghai Qin
Geng Yuan
Linke Guo
Shiwei Liu 0003
Xiaolong Ma

Dynamic Sparse Training (DST) is an effective approach for addressing the substantial training resource requirements posed by the ever-increasing size of the Deep Neural Networks (DNNs). Characterized by its dynamic "train-prune-grow” schedule during training, DST implicitly develops a bi-level structure for training the weights while discovering a subnetwork topology. However, such a structure is consistently overlooked by the current DST algorithms for further optimization opportunities, and these algorithms, on the other hand, solely optimize the weights while determining masks heuristically. In this paper, we extensively study DST algorithms and argue that the training scheme of DST naturally forms a bi-level problem in which the updating of weight and mask is interdependent. Based on this observation, we introduce a novel efficient training framework called BiDST, which for the first time, introduces bi-level optimization methodology into dynamic sparse training domain. Unlike traditional partial-heuristic DST schemes, which suffer from sub-optimal search efficiency for masks and miss the opportunity to fully explore the topological space of neural networks, BiDST excels at discovering excellent sparse patterns by optimizing mask and weight simultaneously, resulting in maximum 2. 62% higher accuracy, 2. 1$\times$ faster execution speed, and 25$\times$ reduced overhead. Code available at https: //github. com/jjsrf/BiDST-ICML2024.

Details

ICLR Conference 2024 Conference Paper

NeurRev: Train Better Sparse Neural Network Practically via Neuron Revitalization

Gen Li 0012
Lu Yin 0006
Jie Ji
Wei Niu 0002
Minghai Qin
Bin Ren 0002
Linke Guo
Shiwei Liu 0003

Dynamic Sparse Training (DST) employs a greedy search mechanism to identify an optimal sparse subnetwork by periodically pruning and growing network connections during training. To guarantee effectiveness, DST algorithms rely on high search frequency, which consequently, requires large learning rate and batch size to enforce stable neuron learning. Such settings demand extreme memory consumption, as well as generating significant system overheads that limit the wide deployment of deep learning-based applications on resource-constraint platforms. To reconcile such, we propose $\underline{Neur}$on $\underline{Rev}$italization framework for DST (NeurRev), based on an innovative finding that dormant neurons exist with the presence of weight sparsity, and cannot be revitalized (i.e., activated for learning) even with high sparse mask search frequency. These dormant neurons produce a large quantity of zeros during training, which contribute relatively little to the outputs of succeeding layers or to the final results. Different from most existing DST algorithms that spare no effort designing weight growing criteria, NeurRev focuses on optimizing the long-neglected pruning part, which awakes dormant neurons by pruning and incurs no additional computation costs. As such, NeurRev advances more effective neuron learning, which not only achieves outperforming accuracy in a variety of networks and datasets, but also promoting a low-cost dynamism at system-level. Systematical evaluations on training speed and system overhead are conducted on the mobile devices, where the proposed NeurRev framework consistently outperforms representative state-of-the-arts. Code will be released.

Details