UCSF brain implant lets people with paralysis speak and gesture through an avatar at the same time
Edward Chang's team decoded concurrent speech and upper-body gestures, a step toward more natural, expressive communication BCIs.
Researchers at the University of California, San Francisco have built a brain-computer interface that turns brain activity into speech and upper-body gestures at the same time. The output drives a full-body virtual avatar. Earlier systems could restore one of these or the other, but not both together.
The study was led by neurosurgeon Edward Chang and published in Nature Neuroscience. It involved three people with severe paralysis of the vocal tract and body, caused by conditions such as ALS or a brainstem stroke. All three still had intact activity in the motor cortex. Two of them controlled the avatar’s speech and gestures successfully.
Key finding
Brain activity during combined speech and gesture looked noticeably different from activity during speech alone or gestures alone. Decoders trained on the combined data did better than separate decoders stitched together. In other words, the brain doesn’t simply add a “speech” signal to a “gesture” signal.
Why it matters
Communication is more than words. Speech BCIs have improved quickly in recent years (see how decoding works), and adding gesture and expression brings them closer to natural conversation. Dr. Chang put it this way: “Conversation is about much more than the words being spoken.”
Learn more: Speech neuroprosthesis · Chang Lab in the Directory