What Memory Is Made Of
The relationship between synesthesia, vivid mental imagery, and memory may offer a view into our unique inner theaters.
"Radiating Affection" from the book, Thought Forms (1901) Credit: Annie Besant and C.W. Leadbeater
“The accident was in April of 2018. There was a handyman on the roof, a couple of stories up. He dropped a pair of shears, a heavy-duty pair of tin snips...I was hit with the rubberized handle.” Callie, a playwright and educator, recalled the accident that led to a severe concussion, “I didn’t want to sit down because I thought if I sat down, I would pass out, and I didn’t want to pass out.” She had relied on her quick memory and vivid imagination to build storylines and dream up characters. She described her mental experience of writing before the incident, “I saw things like a movie in my head. I would see the location...I’d sort of see arcs of movement and three-dimensional sculptures of lines.” The visualizations used to reference plot points are common in people with certain types of synesthesia — a blending of sensory experiences — and those with highly visual imaginations. Due to her accident, her ability to think visually had vanished — she could no longer create or manipulate images in her mind. She had also lost most of her ability to remember basic information like numbers, directions, or names.
Research into memory and synesthesia may provide insight into its connection with the type of episodic visualization Callie had lost. Recall in those with synesthesia and highly visual imaginations are often markedly higher than those without the condition. And, people who lack the capability to make mental images — a condition known as aphantasia — tend to score lower on memory tests. The findings offer a clue into how creative people visualize and remember stories and offer a better understanding of those who don’t.
“There is a continuum, from people who have no visual imagery to people who have very strong and vivid visual imagery,” explained Dr. Beat Meier from the University of Bern, whose research first showed memory advantages in people with synesthesia. “Synesthesia is just one case where we know that people have more vivid imagery.” His research focuses on the brain’s ability to encode information and his interest in synesthesia is due to its unique advantage in that area. He suspects that dual-coding, meaning using more than one reference to encode a memory, such as color, smell, or sound, can increase retention, even when associations are automatic.
People with synesthesia experience a blending of two cognitive states. Often mythologized as a psychedelic experience, it sometimes goes unnoticed by synesthes for a portion of their lives. It can take different forms, some people see colors when they read or hear sound, while others can smell shapes or taste music. People with the condition can also experience more than one type. Spatial-sequence synesthesia can manifest as three-dimensional shapes that relate to numbers, time, or even plot — providing the ability for synesthes to jump between “scenes,” like what Callie described. Associative synesthetes experience everything within the mind’s eye, while projector synesthetes experience them outside of their body, as if they’re wearing a pair of augmented reality goggles.
The condition may arise due to brain cell connections and the proximity of our neurons. “For grapheme-color synesthesia, people have activations in their temporal lobe...in the area in the letter area and the color area; they are quite close together,” explained Meier. As our brains grow normally there may be a pruning among the connections. In synesthesia those connections may be somehow inhibited. “It is possible that kind of an obsessive exposure to these connections is necessary.”
Callie described the contrast between her pre- and post-concussion experience, “I didn’t know there was any other way to experience the world. I was always seeing stories. I’d drive by a house, and I would see a whole movie in that house. I didn’t even know the people living there, but there just be a story. I just had a very, very vivid imagination.” After the head injury, Callie lost that normally rich ability to make images. “There was nothing. It was like this black void. Everything was just flat.” She struggled to navigate ordinarily simple tasks due to her impaired memory, “I had to use GPS to go to the grocery store because I couldn’t remember where I was going.”
“I didn’t know there was any other way to experience the world. I was always seeing stories. I’d drive by a house, and I would see a whole movie in that house.”
Memory and visualization have long been associated. Ancient Greek senators used a technique called “a memory palace” to give speeches. They would use items representing words or phrases and imagine them placed in a familiar space, such as a person’s home. As the observer “walks” through its hallways and rooms, they are able to convert the items back into words and sentences, allowing them to reproduce long orations from memory.
Those with eidetic, or “photographic,” memories often describe a similar experience. The actress Marilu Henner shared her ability to remember events in perfect detail on an episode of 60 Minutes. “It’s like putting in a DVD and it cues up to a certain place. I’m there again. So, I’m looking out from my eyes and seeing things visually as I would have that day,” she said. It’s as if she is able to time travel within the mind’s eye.
Meier shared the story of the artist Stephen Wiltshire, who has an extraordinary gift. “He is able to, when he flies over a city...kind of imprint the structure of a city. Then he goes and draws the city astonishingly well.” His drawings, done from memory, are then rendered in black and white in perfect detail. “He has extremely expert memory for cars and for buildings, and stuff like this…” he said. Notably, Wiltshire is autistic, a condition that typically is associated with poor visual and episodic memory. His process was described by Oliver Sachs who studied Whiltshire as an obsession, with little concern for the context or narrative detail that would normally support a memory.
There is a recent study that explores the relationship between aphantasia and synesthesia, and autism out of the University of Sussex. It found that synesthesia co-occurred across all participants equally in all three conditions. This may mean that synesthesia is not necessary to drive how vivid our mental landscapes are. More likely, the condition is a supplemental tool to enhance memory in already highly visual individuals. When co-occurring with aphantasia the subjects scored higher on memory tests than others with the condition, showing an advantage even when mental images aren’t present.
Meier’s research found that memory advantages in synesthetes seem to be “type-specific.” Color-graphene synesthetes, those would see colors associated within the letters of the alphabet, tend to score highest on memory tests. He’s currently studying chromesthesia, or those who see color associated with music, and how it relates to perfect pitch. “Musicians who have absolute pitch, the prevalence of synesthesia is higher in these people than in the normal population,” he said. Having a second reference point to musical scale may allow the brain to reference notes visually — a huge advantage when identifying pitch out of context.
As Callie’s received treatment and her concussion healed, her memory and visual forms returned. “That ability to imagine came back,” she said. “I came up with the entire timeline, characters, character names, exactly how many scenes there were, the main events...like the resonance objects, the pivotal moments, all of it...I was like, ‘oh, I can imagine!’” Her experience sheds light on the mind’s visual projector and how it relates to our episodic imagination and memory. The pathway of a thought and a visualization may cross brain regions that are usually separated. In some people, these regions are intertwined, leading to extraordinary experiences of perception and thought: three-dimensional sculptures of lines or color enhanced experience — and for others, near perfect memory. It may be indicative of a mind that stages and poses, conducts, and illuminates vs. one that tells its stories in the dark.