Scientists have reconstructed short videos using only brain activity recorded from mice, effectively allowing researchers to recreate what the animals were seeing. The study, led by researchers at University College London (UCL), offers a new way to investigate how the brain transforms visual input into an internal representation of the world.
Published in eLife, the findings could help scientists better understand how visual information is processed in the brain and may eventually make it possible to compare how different species perceive the same surroundings.
Decoding What the Brain Sees
Researchers have spent years trying to understand how the brain interprets signals arriving from the eyes. In human studies, scientists have shown people images and movies while recording brain activity with fMRI, then used those measurements to try to reconstruct visual information down to the level of individual pixels.
The new work takes a different approach. Instead of relying on broader brain imaging signals, the researchers used recordings from individual brain cells in mice. These single-cell measurements can provide a more detailed picture of how visual information is represented in the brain.
Using activity recorded from the visual cortex, the team was able to generate high-quality reconstructions of videos that had been shown to the mice.
Lead author Dr. Joel Bauer (Sainsbury Wellcome Centre at UCL) said: “We wanted to have a better way of investigating how the brain interprets what we see. The current methods of understanding what specific groups of neurons are representing are not very generalizable to situations which haven’t been specifically tested for. And so, we wanted to develop a method that can capture what is being represented in the brain and compare that to reality.”
The researchers are especially interested in the differences between what is physically present in front of an animal and how that information is represented inside the brain. Studying those differences could reveal which visual features the brain emphasizes, changes, or filters out.
Turning Neuron Activity Into Video
To reconstruct the movies, Dr. Bauer and his colleagues used a dynamic neural encoding model originally developed by another research team for the 2023 Sensorium Competition. The model is designed to predict how individual neurons (brain cells) will respond while mice watch movies. It also takes into account other factors, including the animals’ movements and changes in pupil diameter.
The UCL researchers then refined the approach using the same dataset. First, they calculated how the neurons were predicted to behave if the mouse had been looking at a blank screen. They compared that prediction with the neurons’ actual activity while the mouse watched a movie.
The neural activity had been measured with a microscopic imaging method that identifies which individual brain cells are active by detecting localized increases in calcium levels.
Using the difference between predicted and measured activity, an algorithm gradually changed the pixels of an initially blank movie. With each adjustment, the reconstructed video became more similar to the one that had actually been shown to the mouse.
Reconstructing a New 10-Second Movie
After the model had been trained, the researchers gave it a more difficult test. They recorded a mouse’s brain activity while it watched a video that had never been included in the model’s training data.
Using only that neural activity, the system reconstructed a 10-second movie resembling the unseen video.
Dr. Bauer added: “Using this approach, we were able to achieve high-quality reconstructions of 10-second video clips. The accuracy of the reconstructions improved with the inclusion of data from more individual neurons, demonstrating the importance of comprehensive neural data.”
The result suggests that the method was not simply memorizing previously shown videos. Instead, it was able to use patterns of neural activity to infer visual information from a new scene.
Measuring How Closely the Videos Matched
To evaluate the reconstructions, the researchers used a method called pixel correlation, which compares corresponding pixels in the original and reconstructed movies.
The analysis showed only small differences in the timing of the two videos. However, the researchers say there is still considerable room to improve image resolution and the amount of the visual scene that can be reconstructed.
Future work will focus on collecting data that can support sharper reconstructions and cover a larger portion of what the animals are seeing.
Why Our Brains Do Not Simply Record Reality
The researchers now plan to use the technique to investigate a deeper question about vision: how much does the brain’s internal representation differ from the world that is actually in front of us?
Vision is not simply a camera-like recording process. The brain continuously interprets, filters, and modifies incoming sensory information. Understanding exactly where and how those changes happen could reveal important principles about perception.
Dr. Bauer concluded: “We don’t have a perfect representation of the world in our heads. The visual processing pipeline skews and warps our representation in a way that modifies information. This deviation between reality and representations in the brain is not necessarily an error but a feature, reflecting how our minds interpret and augment sensory information. We want to explore how this happens in the brain.”