Computer Vision News - July 2019

Every month, Computer Vision News reviews a research paper from our field. This month we have chosen Deep Image Reconstruction from Human Brain Activity . The authors ( Guohua Shen, Tomoyasu Horikawa, Kei Majima and Yukiyasu Kamitani ) published their article here . One of the most interesting questions today is: when it will be possible to read minds? The recent success of deep neural nets enables to perform new and fascinating tasks. One example for such task is the reconstruction of images from brain signals. Today, many vision groups and brain science groups are devoting their efforts to develop DNN based methods to interpret fMRI scans and generate images using brain activity measurements. The impact of succeeding in such task is huge, as it might help to understand the brain structure, assist the blind, and more. In this article, we will describe one of the best methods to perform this a task, review the method and show some of its results. The frame-work of the task is simple: subjects are inserted into an fMRI machine, then, a sequence of images is shown to them and the fMRI signal is recorded. The images are divided into three classes: natural images, artificial shapes, and alphabetical letters. The fMRI data is represented in a 4-dimensional structure: one dimension is for the time and the other three dimensions are a structural representation of the activity of the brain, represented in voxels. Such dataset contains a set of training images and a set of test images. While this paper performs reconstruction of the images from the brain, also classification of the images into classes is possible. The authors use DNN visual features of a given image based on the VGG-19 network. These features are meant to approximate the hierarchical neural representation of the human visual system. Then, the signal from the fMRI is translated to fit into these visual features. At test time, given the translated signal, an optimization is performed to predict the image which its DNN features are best correlated with the fMRI signal. In this way, a reconstruction of the image is generated. The figure in the next page demonstrates the main idea of the method; we now dive into the details of the paper. 8 Research by Amnon Geifman Computer Vision News Research …when will it be possible to read minds?