AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization

While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities across diverse domains, their application to specialized anomaly detection (AD) remains constrained by domain adaptation challenges. Existing Group Relative Policy Optimization (GRPO) based approaches suffer from two critical limitations: inadequate training data utilization when models produce uniform responses, and insufficient supervision over reasoning processes that encourage immediate binary decisions without deliberative analysis. We propose a comprehensive framework addressing these limitations through two synergistic innovations. First, we introduce a multi-stage deliberative reasoning process that guides models from region identification to focused examination, generating diverse response patterns essential for GRPO optimization while enabling structured supervision over analytical workflows. Second, we develop a fine-grained reward mechanism incorporating classification accuracy and localization supervision, transforming binary feedback into continuous signals that distinguish genuine analytical insight from spurious correctness. Comprehensive evaluation across multiple industrial datasets shows that our method achieves superior accuracy by enabling general-purpose MLLMs to acquire fine-grained visual discrimination for detecting subtle manufacturing defects.

Authors

• Jingyi Liao
• Yongyi Su South China University of Technology
• Rong-Cheng Tu Beijing Institute of Technology
• Zhao Jin
• Wenhao Sun School of Statistics and Mathematics, Zhejiang Gongshang University, Hangzhou, China
• Yiting Li Institute for Infocomm Research (I2R), A*STAR
• Xun Xu Institute for Infocomm Research, A*STAR South China University of Technology
• Dacheng Tao College of Computing and Data Science, Nanyang Technological University
• Xulei Yang Institute for Infocomm Research (I2R), A*STAR

Keywords

No keywords are indexed for this paper.