DIFFA: Large Language Diffusion Models Can Listen and Understand

Recent advances in large language models (LLMs) have shown remarkable capabilities across textual and multimodal domains. In parallel, large language diffusion models have emerged as a promising alternative to the autoregressive paradigm, offering improved controllability, bidirectional context modeling, and robust generation. However, their application to the audio modality remains underexplored. In this work, we introduce DIFFA, the first diffusion-based large audio-language model designed to perform spoken language understanding. DIFFA integrates a frozen diffusion language model with a lightweight dual-adapter architecture that bridges speech understanding and natural language reasoning. We employ a two-stage training pipeline: first, aligning semantic representations via an ASR objective; then, learning instruction-following abilities through synthetic audio-caption pairs automatically generated by prompting LLMs. Despite being trained on only 960 hours of ASR and 127 hours of synthetic instruction data, DIFFA demonstrates competitive performance on major benchmarks, including MMSU, MMAU, and VoiceBench, outperforming several autoregressive open-source baselines. Our results reveal the potential of large language diffusion models for efficient and scalable audio understanding, opening a new direction for speech-driven AI.

Authors

• Jiaming Zhou
• Hongjie Chen
• Shiwan Zhao
• Jian Kang
• Jie Li Shanghai Jiao Tong University Shanghai Jiao Tong University (Wuxi) Blockchain Advanced Research Center
• Enzhi Wang College of Computer Science, Nankai University
• Yujie Guo
• Haoqin Sun College of Computer Science, Nankai University
• Hui Wang
• Aobo Kong College of Computer Science, Nankai University
• Yong Qin
• Xuelong Li Institute of Artificial Intelligence (TeleAI), China Telecom

Keywords

No keywords are indexed for this paper.