Reconstruction of natural images from fMRI using BOLD5000

Abstract

Neural activation decoding has several challenges. Neural activation is different for all people, and measurement methods such as functional magnetic resonance imaging (fMRI) are sensitive to external stimuli. In this paper, we reconstruct visual stimulus using the neural activation measured with fMRIs in four patients. We use an fMRI model to obtain a latent representation vector to guide a Latent Diffusion Model (LDM) to generate high-quality images. The fMRI latent vector trained to be similar to the latent vector in the image gives you general information about the original stimuli, such as color or texture, but retraining the encoding model during LDM training shows better-quality generated images. The final results have consistency between the color, textures, shapes, and, in some cases, semantic context with the original stimulus.