Song Reconstructed from Brain Activity

choi

Daniela Hernandez, Song Reconstructed from Brain Activity. Wall Street Journal, Aug 16, 2023, at page A3
https://www.wsj.com/articles/min ... n-research-a643705f

Note:
(a) The article is free. You may find the lyrics in the Web.
(b) Please read the article online, because it carries
(i) a reconstructed version (an audio) compared with the original, which of course is not in the print;
(ii) an illustration of four panels, whose caption reads in print: "The reconstructions from neural activity (bottom three panels) look visually similar to the original song (top panel).." )The online version is identical, except the source (of the illustration) at the end: "PHOTO: LUDOVIC BELLIER (who is the scientist at the UC Berkeley]."
(c) Here is the scientific report:
Ludovic BELLIER et al, Music Can Be Reconstructed from Human Auditory Cortex Activity Using Nonlinear Decoding Models. PLOS Biology, Aug 15, 2023
https://doi.org/10.1371/journal.pbio.3002176
(i) The envelope behind Bellier's name means he is the corresponding author.
(ii) PLOS Biology
https://en.wikipedia.org/wiki/PLOS_Biology
(iii) The illustration in (b)(ii) is taken from Figure 3 C and D of the report embodied in (c).
(d) If you understand, then there is no need to read a commentary:
F Perry Wilson, MD, MSCE, The Sound of Music Directly From Your Brain. Medscape, Aug 15, 2023.
https://www.medscape.com/viewarticle/995198?form=fpf

Quote:

(i) "I want you to listen to this audio file and see if you can recognize it. * * * Some of you, particularly of my generation, may recognize the characteristic rhythm guitar and harmony of Pink Floyd's 'Another Brick in the Wall.' But you haven't heard it like this before. No one has heard any song like this before because this is the first song reconstructed from the electrical signals in a patient's brain.

(ii) "To investigate the way the brain processes music, researchers, led by Ludovic Bellier at UC Berkley [sic], examined 29 patients undergoing neurosurgery. Their results appear in a new paper in PLOS Biology.

"The patients, who were conscious during the operation, had about 300 electrodes placed directly on the surface of the brain. And then they listened to that all-time classic of rock opera, "Another Brick in the Wall (Part 1)."

"The fact that researchers could reconstruct the song from the electrodes is clearly the coolest part of the study, but it wasn't really its purpose. Rather, it was to identify the regions of the cortex of the brain that process music and the components of music.

* MSCE after MD means Master of Science in Clinical Epidemiology.
* F Perry Wilson, MD, MSCE. Tale School of Medicine, undated
https://medicine.yale.edu/profile/francis-p-wilson/
("Associate Professor of Medicine (Nephrology) and Public Health (Chronic Disease Epidemiology); Director, Clinical and Translational Research Accelerator (CTRA); Course Director, Interpretation of the Medical Literature; Co Director, Human Genetics and Clinical Research Core")

------------WSJ
In a scientific breakthrough, researchers have gotten one step closer to developing devices that say aloud what people are thinking.

With further development, this could give patients who have lost the ability to speak—because of a stroke or other brain injuries—the power to communicate in a way that sounds more natural and less robotic.

Neuroscientists at the University of California, Berkeley reconstructed Pink Floyd’s classic rock song “Another Brick in the Wall, Part 1” using recordings of the brain activity of 29 patients who heard the song while undergoing brain surgery. Though less polished than the band’s version, the reconstructed tune was recognizable.

The study was published Tuesday in the journal PLOS Biology.

The reconstruction of complex musical elements from brain-activity patterns advances the development of therapeutic technologies to help patients regain the ability to communicate.

“It’s very exciting. What they’re showing is that they can get pretty high performance with a relatively low amount of data,” said Dr. Edward Chang, a neurosurgeon at the University of California, San Francisco’s Weill Institute for Neurosciences who studies music and language perception.

Chang wasn’t involved in the study but has previously worked with the study authors.

As computer models and artificial intelligence have become more sophisticated, similar studies using brain-computer interfaces—devices that interact with the brain—have re-created images and produced simple language from neural data.

Last year, Elon Musk’s Neuralink, for instance, showed a video of “telepathic typing” from a monkey with a Neuralink brain implant, which contains threadlike electrodes—the equivalent of microphones for neural activity—that penetrate the brain. The animal wasn’t typing into a keyboard but was able to move a cursor to images of letters.

In the new study, researchers took recordings from different regions of the brain using a grid-like pad studded with electrodes placed on the organ’s surface as patients underwent surgery for treatment-resistant epilepsy. Each heard the Pink Floyd song a single time.

The researchers then homed in on some brain areas, like the superior temporal gyrus in the auditory cortex, which is responsible for processing sounds. The activity there—monitored by only a handful of electrodes—tracked most closely with the hit song. The team found they could largely tune out the rest of the data and still get recognizable, if somewhat garbled, playbacks.

Visual versions of the original and reconstructed songs—which look like a mashup of a heat map and an audio waveform—also resembled each other. That was the case for individual patients and all 29 in aggregate.  

“Some of the richness of the original audio frequencies isn’t there in the reconstructions, but it looks like a pretty good approximation, even visually,” said Dr. AZA Allsop, a musician and assistant professor of psychiatry at Yale University and Howard University who wasn’t involved in the study.

“It goes to stir the imagination in terms of what this could enable” in the long term, he added, including the ability to compose music directly from neural recordings.

The researchers’ goal is to leverage the technology to develop neural prosthetics that can give patients back the ability to speak more naturally, according to Dr. Robert Knight, a University of California, Berkeley neuroscientist and an author on the new study.

Currently, “the output of these devices has a monotonic, kind of robotic quality” he said. “Music, given its strong emotional and rhythmic components, would allow us to add an effectual component to [computer-generated] speech.”

The new algorithms also were able to sing back partial vocals. The playback of one patient’s brain activity, for example, contained a few discernible words, including “all,” “was” and “just a brick,” according to the study.

The researchers chose “Another Brick in the Wall, Part 1” because they are fans of Pink Floyd, the English rock band that recorded the song in the 1970s, Knight said. Part 1 is the first segment of a three-part composition and is less well-known than Part 2, the chart-topping single most people recall.

“It’s something that people are kind of familiar with, but not everyone has heard it” so they might not have too many memories associated with it, said Ludovic Bellier, a computational neuroscientist and another of the study’s authors. “It’s kind of in a sweet spot of familiarity.”   

That sweet spot likely differs by age group, personal preference and culture, said Allsop. The algorithms used to reconstruct the Pink Floyd jam will likely need to be retrained on other musical genres with different melodies and rhythms, such as jazz or reggaeton, according to Chang.

Another consideration—recalled in Pink Floyd’s lyric “we don’t need no thought control”—speaks to concerns some neuroethicists, neuroscientists and legal experts have about mental privacy.

Algorithms already can construct pretty accurate models of a person’s preferences based on physical activities, including typing search terms or tapping to “like” a photo. While there are high hopes that reconstructing words and music from neural activity could enable applications that promote mental health and social connection, reading and interpreting thoughts is regarded as the next privacy frontier.

“This could mark a completely different reach inside the mind,” said Lucy Nalbach Tournas, a board member of the Institute of Neuroethics. “It speaks to how powerful music is.”

The ability of music to move people and spark their curiosity is something that Knight quickly realized after doing this work.

“The media hasn’t really been interested in anything I’ve done in the last 40 years, but it’s all been made up for by Pink Floyd,” he said. “It’s gotten a little crazy in terms of people’s interest, which is good.”

Write to Daniela Hernandez at daniela.hernandez@wsj.com