By Frederick d’Oleire Uquillas, Science Communications Fellow for the AI Lab

Imagine lying in an MRI scanner while looking at a picture of a dog. A computer analyzes your brain activity and, seconds later, generates a rough image of that dog.

Not by reading the picture on the screen, but by reading your brain.

That’s the premise behind a new study, “Real-time Reconstruction of Human Visual Perception from fMRI,” in which researchers from Princeton University, supervised by Professor Ken Norman, demonstrate that it’s possible to reconstruct images someone is looking at using only brain activity measured in an MRI scanner – and to do it within seconds.

It’s not perfect mind reading, but it’s close enough to make you do a double take.

The big idea: Translating brain activity into pictures

When you look at an image, say a lighthouse, a person skiing, or a plate of food, your visual cortex produces a unique pattern of activity. Functional MRI (fMRI) measures this activity indirectly through changes in blood oxygen levels across the brain.

For decades, neuroscientists have tried to decode these patterns. The question has always been simple to ask and very hard to answer: If we know the brain activity pattern, can we figure out what someone is seeing?

Recent advances in artificial intelligence have made that question much more tractable. Instead of trying to map brain activity directly to pixels, modern approaches translate brain signals into a high-level representation of the image. Essentially a semantic description of its visual features. Once you have that representation, generative AI can turn it back into an image.

This new study shows that such decoding can now happen in real time.

The experiment

The experiment itself is surprisingly straightforward. A participant laid inside a 3-Tesla MRI scanner while looking at hundreds of natural images – things like animals, people, and outdoor scenes. Each image appeared for a few seconds while the scanner measured activity across visual regions of the brain.

The researchers first collected about an hour of training data from the participant. During that phase, the AI system learned how that individual’s brain responded to different images. Once trained, the model could be used in a second scanning session to reconstruct what the participant was seeing.

What the reconstructions look like

The reconstructions aren’t photographic copies of the original images. Instead, they look more like AI-generated approximations.

If the participant saw a skier wearing a red jacket on a snowy mountain, the reconstruction might show a person-shaped figure in red surrounded by snowy terrain. If the participant saw animals grazing in a field, the reconstruction might produce a scene containing animal-like shapes in a grassy landscape.

They’re fuzzy and imperfect, but unmistakably related to the original image. The system captures the broad structure and meaning of the scene even when the details are not exact.

Quantitatively, the reconstructions perform significantly better than chance according to multiple image similarity metrics used in the study.

The real breakthrough: speed

Earlier methods for reconstructing images from fMRI data could take hours, or even days.

This system works much, much faster.

In the fastest configuration, the researchers were able to produce image reconstructions about 15 seconds after the image was viewed.

That might not sound instantaneous, but for fMRI it is remarkably fast. The delay mostly comes from biology rather than computing power: The blood-oxygen signal measured by fMRI naturally peaks several seconds after a stimulus appears and can take up to 20 seconds to fully dissipate.

Even with that constraint, the system processes the brain data and runs the AI models in just a few seconds once the signal arrives.

Why this is technically difficult

Two fundamental challenges make real-time brain decoding hard.

The first challenge is that fMRI signals are noisy and indirect. Instead of measuring neurons directly, the scanner measures blood flow changes associated with neural activity. That signal unfolds slowly and can overlap across multiple stimuli.

The second challenge is computational. The AI models used to reconstruct images are large and complex, with hundreds of millions of parameters. Running them quickly enough to keep up with incoming brain data requires careful engineering and substantial computing resources.

The researchers addressed this by combining a cloud-based real-time fMRI framework with a streamlined version of the MindEye2 architecture, allowing the system to analyze brain activity and generate reconstructions within seconds.

Why we should care

Beyond the “mind reading” headlines, the real significance lies in what this technology enables.

Real-time decoding opens the door to closed-loop neuroscience experiments, where brain activity can be analyzed and fed back to the participant while the experiment is still happening. For example, researchers could monitor how the brain represents a visual stimulus and then adjust the stimulus based on that representation.

There are also potential clinical applications. Real-time decoding might eventually help researchers understand how people with depression, anxiety, or other conditions interpret emotional images or stimuli. (A 2021 study from a team that included Princeton researchers found that people with depression get better at ignoring faces with negative expressions when they’re given real-time feedback on their brain activity.) In principle, the reconstructed image could reveal subtle biases in how someone perceives the world.

 A quick reality check

Before we start worrying about someone reading our thoughts on the subway, it’s worth emphasizing the limitations.

The system requires a participant to lie inside a large and very expensive MRI scanner. It also requires personalized training data for each individual before decoding can work reliably. And most importantly, it reconstructs visual perception, not arbitrary thoughts or memories.

So this isn’t a universal mind reader. It’s more like a sophisticated translator between brain activity and visual representations.

Still, it’s an impressive step.

The bigger picture

For decades, neuroscientists have dreamed of decoding the contents of perception from brain activity. What makes this moment different is the convergence of two technological revolutions: Large-scale neuroimaging datasets, and powerful generative AI models.

Together, they allow scientists to map brain activity into rich visual feature spaces that AI already understands.

Once that mapping exists, generating an image becomes surprisingly straightforward.

The takeaway

The images reconstructed in this study aren’t perfect. But the fact that they exist at all, and can appear within seconds of seeing a stimulus, is remarkable.

It suggests that the gap between brain signals and subjective experience may be smaller than we once thought.

We’re not reading minds yet.

But we’re starting to sketch them.

The research was funded by Princeton’s Office of the Dean for Research Innovation Fund for New Industrial Collaborations, in partnership with Sophont. Early stages of the research were supported by the National Institute of Mental Health.

Curious to learn more?

Check out the full paper here.