The process for creating video from brain scans was recently reported by experts from the National University of Singapore and the Chinese University of Hong Kong. There is a copy of the study available on the public arXiv preprint server.
To create high-quality videos with little to no jerkiness, researchers Jiaxin Qing, Zijiao Chen, and Juan Helen Zhou used a technique called functional magnetic resonance imaging (fMRI) and coupled data retrieved through imaging with the deep learning model Stable Diffusion.
Commonly used for image generation, AI-assisted Stable Diffusion has been reported to successfully recreate still imagery gleaned from brain activity in recent months. However, as reported by Qing and company, the challenge of retrieving continuous visual images is significant.
fMRI typically uses blood oxygenation level-dependent (BOLD) signals to take pictures of brain activity every few seconds. Because of this, the video quality would suffer. There are 30 frames per second as the norm for video recording.
“We show that high-quality videos of arbitrary frame rates can be reconstructed with Mind-Video using adversarial guidance. The recovered videos were evaluated with various semantic and pixel-level metrics,” the study’s authors explained in their report.
“We achieved an average accuracy of 85% in semantic classification tasks and 0.19 in structural similarity index (SSIM), outperforming the previous state-of-the-art by 45%. We also show that our model is biologically plausible and interpretable, reflecting established physiological processes.”
