Brain implant decodes speech, gestures for paralyzed patients

Clinical takeaway: Brain-computer interfaces are advancing beyond text. A single implant decoded attempted speech and gestures together, though this remains an early feasibility result. 

Severe paralysis from brainstem stroke or ALS can take away speech along with the gestures that carry meaning beside it, from a wave to a raised hand. The current standard for these patients, eye-tracking software that spells out text, is slow, physically draining, and limited to words on a screen. What's lost is not just vocabulary but the ability to say something and show something in the same moment, the way conversation naturally works. 

Implanted devices have started to close part of that gap, translating attempted speech into text or a synthesized voice in some studies and attempted hand movements into computer commands in others. Each of those demonstrations handled one channel in isolation. Natural conversation runs both at once, and it hadn't been shown that a single implant could read concurrent speech and gesture signals without confusing one for the other. A small feasibility study now answers that question in early form, showing that one sensor array can drive both, and explaining why systems built the old way would have stumbled. 

In real-time testing, one participant controlled an avatar that displayed his words and gestured at once, producing the correct output on 70% of speech attempts and 66% of gesture attempts. That was across 10 phrases and 10 gestures where chance alone would have been about 9%. In a conversational task, answering spoken prompts rather than copying cued targets, that participant's accuracy reached 75% for speech and 85% for gestures. A second participant with a smaller vocabulary performed at or near 100% accuracy. 

Simultaneous decoding required a training fix. Brain activity during combined attempts differed from activity during speech or gestures alone, so decoders trained only on one behavior at a time lost accuracy when both were tried together. Training on a mix of solo and combined attempts restored it. The decoders also misfired at first, with the speech decoder producing words during gesture-only attempts, or the reverse, on up to 76% of trials. Teaching each decoder to ignore the other's signals cut those errors to zero. 

The findings come from a phase 1 feasibility trial in which three adults with severe paralysis (two from brainstem stroke, one from ALS) were implanted with a 253-electrode sensor array over the brain's sensorimotor cortex. Participants attempted phrases and gestures, alone and together, while the array recorded the signals used to train and test the decoders. One participant withdrew before the avatar work, so the simultaneous results involve two people; the system remains wired and investigational. Most in-depth testing involved the ALS participant; the second was evaluated mostly offline. 

The bottleneck is hardware, as much as it is the software. The system still runs through a wired connection between the implant and external computers, and the research team's next step is testing a fully implantable, wireless version built for long-term use. Scaling is the other open engineering problem. Practical conversation needs far more than ten phrases and ten gestures; larger vocabularies will demand faster decoding and broader training data. 

"These promising results give me hope that in the future, patients with severe paralysis will be able to recapture the holistic nature of human communication," said Debara Tucci, MD, director of the National Institute on Deafness and Other Communication Disorders. 

Source: Brosler SC, et al. (2026 Sep 14) Nat Neurosci. Simultaneous speech and gesture decoding for multimodal communication in paralysis