OmniScale: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo

Recent advances in large language models (LLMs) have driven impressive progress in omni-modal understanding and generation. However, training omni-modal LLMs remains a significant challenge due to the heterogeneous model architectures required to process diverse modalities, necessitating sophisticated system design for efficient large-scale training. Existing frameworks typically entangle model definition with parallel logic, incurring limited scalability and substantial engineering overhead for end-to-end omni-modal training. We present OmniScale, a modular and efficient training framework to accelerate the development of omni-modal LLMs. OmniScale introduces model-centric distributed recipes that decouples communication from computation, enabling efficient 3D parallelism on omni-modal LLMs. OmniScale also features a flexible configuration interface supporting seamless integration of new modalities with minimal code change. Using OmniScale, a omni-modal mixture-of-experts (MoE) model with 30B parameters can be trained with over 2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D parallelism on 128 GPUs, showcasing its superior efficiency and scalability for training large omni-modal LLMs.

Authors

• Qianli Ma South China University of Technology
• Yaowei Zheng
• Zhelun Shi
• Zhongkai Zhao
• Bin Jia ByteDance Seed
• Ziyue Huang ByteDance Seed
• Zhiqi Lin
• Youjie Li ByteDance Seed
• Jiacheng Yang
• Yanghua Peng ByteDance Seed
• Zhi Zhang The Hong Kong Polytechnic University
• Xin Liu

Keywords

No keywords are indexed for this paper.