Controllable Mind Visual Diffusion Model

Brain signal visualization has emerged as an active research area, serving as a critical interface between the human visual system and computer vision models. Diffusion-based methods have recently shown promise in analyzing functional magnetic resonance imaging (fMRI) data, including the reconstruction of high-quality images consistent with original visual stimuli. Nonetheless, it remains a critical challenge to effectively harness the semantic and silhouette information extracted from brain signals. In this paper, we propose a novel approach, termed as Controllable Mind Visual Diffusion Model (CMVDM). Specifically, CMVDM first extracts semantic and silhouette information from fMRI data using attribute alignment and assistant networks. Then, a control model is introduced in conjunction with a residual block to fully exploit the extracted information for image synthesis, generating high-quality images that closely resemble the original visual stimuli in both semantic content and silhouette characteristics. Through extensive experimentation, we demonstrate that CMVDM outperforms existing state-of-the-art methods both qualitatively and quantitatively. Our code is available at https://github.com/zengbohan0217/CMVDM.

Authors

• Bohan Zeng Beihang University
• Shanglin Li Beihang University
• Xuhui Liu Beihang University
• Sicheng Gao Beihang University
• Xiaolong Jiang Xiaohongshu Inc.
• Xu Tang Xiaohongshu
• Yao Hu Xiaohongshu
• Jianzhuang Liu Shenzhen Institute of Advanced Technology
• Baochang Zhang Hangzhou Research Institute, School of Artificial Intelligence, Beihang University, China Nanchang Institute of Technology, Nanchang, China

Keywords

• CV: Computational Photography, Image & Video Synthesis