A Brain Implant That Lets Paralyzed People Speak and Gesture at Once

A single brain implant decoded attempted speech and attempted hand gestures at the same time in people with severe paralysis, a team at the University of California, San Francisco, reported Sept. 14 in Nature Neuroscience.

Speech-restoring implants give a paralyzed user words, but not the movement that normally travels with them. In the paper, first author Samantha C. Brosler, senior author Edward F. Chang and colleagues note that speech is frequently accompanied by gestures, and describe the work as a step toward implants that support more natural communication.

Three volunteers in a UCSF clinical trial took part, paralyzed by a brainstem stroke or by ALS. Each has a grid of electrodes resting on the surface of the sensorimotor cortex, the brain region that drives movement.

The simultaneous task was run with two of them. The participant with ALS worked from a vocabulary of 10 phrases and 10 gestures, a set the authors describe as a proof of concept. With a speech decoder and a gesture decoder running in parallel in real time, the system picked the right phrase 70.0% of the time and the right gesture 66.0%, against a chance level of 9.1%. Decoded phrases appeared as text while decoded gestures animated a personalized full-body avatar on screen.

Decoders built only from sessions of speech alone, or of gestures alone, did not carry over when the two were attempted together. Adding simultaneous attempts to the training data restored accuracy in both settings. The models also handled phrase-and-gesture pairings they had never been trained on together about as well as pairings they had, which the authors call encouraging for scaling to larger vocabularies.

The paper is open access and the analysis code is posted on GitHub. The neural recordings stay under restricted access, as set by the clinical trial protocol, and can be shared with researchers at other institutions on request.