SFN 2026 Abstracts
SFN 2026 Abstracts
Come visit us at SFN to hear about our current research. Please see below for our accepted abstracts in the Late Breaking Section (Tues., Nov. 17, 2026 4 - 5 p.m.).
M. Mender1, J. Costello1, P. Pinchi1, M.S. Willsey1
1University of Michigan, Ann Arbor, MI, USA
Most high-performance intracortical brain-computer interfaces (iBCIs) require daily calibration because the relationship between neural recordings and movement changes across days. While a stable neural representation, i.e. a neural manifold, of movement can be retrospectively defined by non-linear dynamical models, these models have not been widely implemented in real-time as a ‘decoder’ to translate neural activity into control signals. Instead, high performance real-time iBCIs have been achieved with less complex models (e.g., Kalman filters, temporally convolved neural networks, and LSTMs) that can be trained quickly. Here we demonstrate stable prospective finger-movement predictions with a lightweight LSTM by training on velocity labels derived from a stable cross-session neural representation.
We studied this in a public non-human primate dataset containing 156 sessions spanning 1200 days with recorded intracortical activity in the precentral gyrus during movements of two finger groups. Using Latent Factor Analysis via Dynamical Systems (LFADS) with session-specific input and output layers, we first identified a consistent neural manifold for two-finger movements. When applied to unseen prospective days, standard LFADS had a 36.8% decrease in velocity decoding performance from factors. We therefore adapted LFADS to use a shared read-in layer with session-specific output layers which enabled projecting neural activity from prospective sessions onto the stable manifold. This improved prospective decoding by 43.9% relative to standard LFADS (only 7.7% decrease from training sessions).
We then made two innovations to the LSTM: adding a neural reconstruction loss and training on velocities from LFADS predictions. We hypothesized that using velocities predicted from this stable neural manifold would enable LSTM decoders to learn consistent behavioral information despite neural instabilities. We trained an LSTM with velocities predicted by LFADS factors over 10 sessions and compared this with an LSTM trained on true velocity labels from those sessions. This new method, denoted the ‘stable label method (SLM)’, achieved a 10.2% increase in prospective performance relative to an LSTM trained on recorded velocities (p=0.01). Additional performance gains could also be achieved with unsupervised fine-tuning using SLM on prospective sessions.
By separating the stable representation learning from real-time decoding, SLM provides a blueprint for leveraging more historical data while retaining a model with rapid training and evaluation. This provides a path towards stable high-performance, low-calibration, multi-effector BCIs.
Adapting pretrained ASR models for brain-to-text decoding
J. Costello1, P. Pinchi1, M.S. Willsey1
1University of Michigan, Ann Arbor, MI, USA
Intracortical speech brain-computer interfaces (BCIs) can restore communication by decoding attempted speech from neural activity. Many brain-to-text systems use a cascaded approach in which neural activity is first decoded into phonemes, followed by a language-model to predict words. Alternatively, end-to-end (E2E) models predict words directly, allowing them to learn abstract latent representations potentially more optimal than phonemes. E2E models have also been pretrained on diverse automatic speech recognition (ASR) datasets and scale well with increasing data. Here, we adapted the Whisper ASR model (Radford et al. 2022), a transformer encoder-decoder architecture, for brain-to-text decoding, and investigated whether audio pretraining transfers to neural inputs and whether E2E models provide advantages over phoneme-based approaches.
We used publicly available recordings from BrainGate2 Clinical Trial participant T15 (Card et al. 2024) who has 256 electrodes across four microelectrode MultiPort Arrays (Blackrock Neurotech) in precentral gyrus. Threshold crossings and spike-band power features were recorded during 10,948 attempted-speech sentences. Neural features were passed to the Whisper encoder, which was either (1) trained with a phoneme CTC head and decoded by a 5-gram language model (cascaded) or (2) paired through cross-attention with Whisper's autoregressive decoder and trained with a word cross-entropy (CE) objective (E2E). The two models were otherwise identical, giving a controlled comparison of language modeling strategy and loss. Accuracy was evaluated with word error rate (WER).
We found that pretraining on large scale audio data significantly improved E2E WER (14% vs 37% without) but not the cascaded model. However, cascaded decoding still achieved lower WER (8%) than E2E (14%), consistent with prior work. CTC training required substantially different hyperparameters than E2E, and a combined CE and CTC loss gave no benefit over either alone, suggesting the two objectives may have competing gradients and that the best way to combine them is yet to be determined. The E2E model also showed a sharp threshold for data and input channels, requiring >2000 sentences or >64 channels to achieve <100% WER; above it, E2E improved more efficiently with data (-11.8% WER per doubling vs -5.4% cascaded).
These results suggest E2E models can benefit from audio pretraining but may not yet outperform search-based language models at current data scales. We are actively exploring data scaling considerations, how results apply across a variety of publicly released datasets, and methods to better combine CTC and CE into the loss function.