本文的第一作者为硕士生柴光洁，题目为“Self-supervised learning guided cross-modal visual reconstruction of fMRI”。

ABSTRACT  Visual stimulation will induce specific neural activity in the visual area of the brain, these neural activities can be captured by functional magnetic resonance imaging (fMRI) and other neuroimaging methods. We may consider the visual stimulus image and the measured fMRI as different representations of the same visual perception in two modalities, the process of reconstructing visual image based on neuroimaging data is called cross-modal visual reconstruction. When the common cross-modal reconstruction model extracts the implicit representation from a single modality, the mapping relationship between the modes is not fully utilized, so it is difficult for the conventional deep network model to reconstruct stimulus images accurately. This paper proposes a self-supervised learning guided cross-modal visual reconstruction model, which can make full use of the data of two modalities, and  use the method of mutual supervision and guidance to accurately reconstruct the visual image from fMRI data. The experimental results show that the proposed method achieves or even exceeds the reconstruction performance of advanced methods on all three public data sets.

Key words  Self-supervised learning, cross-modal, visual reconstruction, fMRI

摘要  视觉刺激会引发大脑视觉脑区的特异性神经活动，通过功能磁共振成像等神经影像方法可以捕捉到这些神经活动。我们不妨将视觉刺激图像和测量到的神经活动，看作同一种视觉感知在两种模态下的不同表现形式。那么，基于神经影像数据，重建出被试所看到的刺激图像的过程，可以称之为跨模态视觉重建。常见的跨模态重建模型从单个模态中提取隐含表征时，没有充分利用模态间的映射关系，所以普通的深度网络模型很难重建出准确的刺激图像。本文提出了一种自监督学习引导的跨模态视觉重建模型，可以充分利用两种模态的数据，采用自相监督与引导的方式，从大脑的功能磁共振成像数据中精确地重建出引发大脑神经元响应的视觉图像。实验结果表明，本文提出的方法在三个公开数据集上均达到或超越了前沿方法的重建效果。

关键字  自监督学习,跨模态,视觉重建,功能磁共振成像