New brain implant translates thoughts from handwriting into text

Elon Musk’s Neuralink has been make waves on the technological side of neural implants, but that has yet to show how we might actually use implants. For now, the demonstration of the promise of implants remains in the hands of the academic community.

This week, this community provided a impressive example of the promise of neural implants. Using an implant, a paralyzed individual managed to type around 90 characters per minute just by imagining he was writing those characters by hand.

Previous attempts to provide typing skills to paralyzed people via implants have involved giving subjects a virtual keyboard and letting them maneuver a cursor with their minds. The process is efficient but slow, and requires the user’s full attention, as the subject must follow the progress of the cursor and determine when to perform the equivalent of pressing a key. It also forces the user to spend time learning how to control the system.

But there are other possible ways to get characters out of the brain and place them on the page. Somewhere in our writing thought process, we form the intention to use a specific character, and using an implant to follow that intention could potentially work. Unfortunately, the process is not particularly well understood.

Downstream from this intention, a decision is transmitted to the motor cortex, where it translates into actions. Again, there is an intention stage, where the motor cortex determines that it will form the letter (by typing or writing, for example), which is then translated into specific muscle movements needed to perform the action. These processes are much better understood and this is what the research team has targeted for their new work.

Specifically, the researchers placed two implants in the premotor cortex of a paralyzed person. It is believed that this area is involved in the formation of intentions to perform movements. Catching these intentions is much more likely to produce a clear signal than capturing the movements themselves, which are likely to be complex (any movement involves multiple muscles) and depend on the context (where your hand is in relation to the page on which you are writing, etc.).

With the implants in the right place, the researchers asked the participant to imagine writing letters on a page and recording neural activity as he did so.

In total, there were approximately 200 electrodes in the participant’s premotor cortex. Not all of them were informative for letter writing. But for those who were, the authors performed principal component analysis, which identified the characteristics of neural recordings that differed the most when various letters were imagined. By converting these recordings to a two-dimensional plot, it was evident that the activity observed when writing a single character was always clustered. And physically similar characters –p and b, for example, or h, not, and r: Clusters formed close to each other.

(The researchers also asked the participant to make punctuation marks such as a comma and question mark and used a> to denote a space and a tilde for a period.)

Overall, the researchers found they could decipher appropriateness with just over 94% accuracy, but the system required relatively slow analysis after recording neural data. To make things work in real time, the researchers trained a recurrent neural network to estimate the probability of a signal matching each letter.

Despite working with a relatively small amount of data (only 242 character sentences), the system performed remarkably well. The time lag between the thought and a character appearing on the screen was about half a second, and the participant was able to produce around 90 characters per minute, easily surpassing the previous record for implant typing, which was of about 25 characters per minute. The raw error rate was around 5% and applying a system such as automatic keystroke correction could reduce the error rate to 1%.

The tests were all done with prepared sentences. Once the system was validated, however, the researchers asked the participant to type free-form responses to the questions. Here the speed decreased a bit (75 characters per minute) and the errors increased by 2% after autocorrection, but the system was still working.

Leave a Reply

Your email address will not be published. Required fields are marked *