Transfer Learning for Covert Speech Classification Using EEG Hilbert Envelope and Temporal Fine Structure

DURAISAMY, Saravanakumar; DUBIEL, Mateusz; REKRUT, Maurice; LEIVA, Luis A.

doi:10.1109/ICASSP49660.2025.10888880

Download

Paper published in a book (Scientific congresses, symposiums and conference proceedings)

Transfer Learning for Covert Speech Classification Using EEG Hilbert Envelope and Temporal Fine Structure

DURAISAMY, Saravanakumar; DUBIEL, Mateusz; REKRUT, Maurice et al.

2025 • In Proceedings of ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Peer reviewed

Permalink
https://hdl.handle.net/10993/64121

DOI
10.1109/ICASSP49660.2025.10888880

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

silent_speech_icassp25.pdf

Author preprint (994.98 kB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Speech BCI; Electroencephalography; Hilbert Envelope; Transfer Learning; Signal Processing

Abstract :

[en] Brain-Computer Interfaces (BCIs) can decode imagined speech from neural activity. However, these systems typically require extensive training sessions where participants imaginedly repeat words, leading to mental fatigue and difficulties identifying the onset of words, especially when imagining sequences of words. This paper addresses these challenges by transferring a classifier trained in overt speech data to covert speech classification. We used electroencephalogram (EEG) features derived from the Hilbert envelope and temporal fine structure, and used them to train a bidirectional long-short-term memory (BiLSTM) model for classification. Our method reduces the burden of extensive training and achieves state-of-the-art classification accuracy: 86.44% for overt speech and 79.82% for covert speech using the overt speech classifier.

Disciplines :

Computer science

Author, co-author :

DURAISAMY, Saravanakumar ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

DUBIEL, Mateusz ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

REKRUT, Maurice; German Research Center for Artificial Intelligence (DFKI)

LEIVA, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

Transfer Learning for Covert Speech Classification Using EEG Hilbert Envelope and Temporal Fine Structure

Publication date :

06 April 2025

Event name :

(ICASSP 2025): 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing

Event place :

Hyderabad, India

Event date :

from 6 to 11 April 2025

Audience :

International

Main work title :

Proceedings of ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Publisher :

Institute of Electrical and Electronics Engineers (IEEE)

Peer reviewed :

Peer reviewed

Focus Area :

Computational Sciences

European Projects :

HE - 101071147 - SYMBIOTIK - Context-aware adaptive visualizations for critical decision making

FnR Project :

FNR15722813 - Brainsourcing For Affective Attention Estimation, 2021 (01/02/2022-31/01/2025) - Luis Leiva

Funders :

UE - Union Européenne

Funding text :

Research supported by the Horizon 2020 FET program of the European Union through the ERA-NET Cofund funding (grant CHIST-ERA-20-BCI-001) and the Pathfinder program of the European Innovation Council (SYMBIOTIK project, grant 101071147). Rekrut’s work is supported by the German Federal Ministry of Education and Research (grants 01IS12050 and 01IS23073).

Available on ORBilu :

since 06 February 2025

Statistics

Number of views

193 (18 by Unilu)

Number of downloads

109 (5 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

J. Freitas, A. Teixeira, M. S. Dias, S. Silva, et al., An introduction to silent speech interfaces, Springer, 2017.
B. Denby, T. Schultz, K. Honda, T. Hueber, J. M. Gilbert, and J. S. Brumberg, “Silent speech interfaces, ” Speech Communication, vol. 52, no. 4, pp. 270-287, 2010.
A. Kapur, S. Kapur, and P. Maes, “Alterego: A personalized wearable silent speech interface, ” in Proceedings of the 23rd International Conference on Intelligent User Interfaces, 2018, pp. 43-53.
N. Kimura, M. Kono, and J. Rekimoto, “Sottovoce: An ultrasound imaging-based silent speech interaction using deep neural networks, ” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1-11.
L. Wang, X. Zhang, X. Zhong, and Y. Zhang, “Analysis and classification of speech imagery eeg for bci, ” Biomedical signal processing and control, vol. 8, no. 6, pp. 901-908, 2013.
J. A. Gonzalez-Lopez, A. Gomez-Alanis, J. M. M. Doñas, J. L. Pérez-Córdoba, and A. M. Gomez, “Silent speech interfaces for speech restoration: A review, ” IEEE access, vol. 8, pp. 177995-178021, 2020.
J. S. Brumberg, A. Nieto-Castanon, P. R. Kennedy, and F. H. Guenther, “Brain-computer interfaces for speech communication, ” Speech communication, vol. 52, no. 4, pp. 367-379, 2010.
J. S. Brumberg, D. J. Krusienski, S. Chakrabarti, A. Gunduz, P. Brunner, A. L. Ritaccio, and G. Schalk, “Spatio-temporal progression of cortical activity related to continuous overt and covert speech production in a reading task, ” PloS one, vol. 11, no. 11, pp. e0166872, 2016.
T. Proix, J. Delgado Saa, A. Christen, S. Martin, B. N. Pasley, R. T. Knight, X. Tian, D. Poeppel, W. K. Doyle, O. Devinsky, et al., “Imagined speech can be decoded from low-and cross-frequency intracranial eeg features, ” Nature communications, vol. 13, no. 1, pp. 48, 2022.
S. Luo, Q. Rabbani, and N. E. Crone, “Brain-computer interface: applications to speech decoding and synthesis to augment communication, ” Neurotherapeutics, vol. 19, no. 1, pp. 263-273, 2023.
D. Lopez-Bernal, D. Balderas, P. Ponce, and A. Molina, “A state-of-the-art review of eeg-based imagined speech decoding, ” Frontiers in human neuroscience, vol. 16, pp. 867281, 2022.
A. S. S. Reddy and R. B. Pachori, “Multivariate dynamic mode decomposition for automatic imagined speech recognition using multichannel eeg signals, ” IEEE Sensors Letters, 2024.
M. Rekrut, A. M. Selim, and A. Krüger, “Improving silent speech bci training procedures through transfer from overt to silent speech, ” in 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2022, pp. 2650-2656.
J. S. García-Salinas, L. Villaseñor-Pineda, C. A. Reyes-García, and A. A. Torres-García, “Transfer learning in imagined speech eeg-based bcis, ” Biomedical Signal Processing and Control, vol. 50, pp. 151-157, 2019.
S.-H. Lee, M. Lee, and S.-W. Lee, “Eeg representations of spatial and temporal features in imagined speech and overt speech, ” in Pattern Recognition: 5th Asian Conference, ACPR 2019, Auckland, New Zealand, November 26-29, 2019, Revised Selected Papers, Part II 5. Springer, 2020, pp. 387-400.
H. Watanabe, H. Tanaka, S. Sakti, and S. Nakamura, “Synchronization between overt speech envelope and eeg oscillations during imagined speech, ” Neuroscience research, vol. 153, pp. 48-55, 2020.
S. Komeiji, T. Mitsuhashi, Y. Iimura, H. Suzuki, H. Sugano, K. Shinoda, and T. Tanaka, “Feasibility of decoding covert speech in ecog with a transformer trained on overt speech, ” Scientific Reports, vol. 14, no. 1, pp. 11491, 2024.
A. Mohamed Selim, M. Rekrut, M. Barz, and D. Sonntag, “Speech imagery bci training using game with a purpose, ” in Proceedings of the 2024 International Conference on Advanced Visual Interfaces, 2024, pp. 1-5.
J. Iriarte, E. Urrestarazu, M. Valencia, M. Alegre, A. Malanda, C. Viteri, and J. Artieda, “Independent component analysis as a tool to eliminate artifacts in eeg: a quantitative study, ” Journal of clinical neurophysiology, vol. 20, no. 4, pp. 249-257, 2003.
D. Hilbert, “Fundamentals of a general theory of linear integral equations, ” in Integral equations and equations with infinitely many unknowns, pp. 6-169. Springer, 1989.
L. He and V. Dellwo, “A praat-based algorithm to extract the amplitude envelope and temporal fine structure using the hilbert transform, ” ISCA, 2016.
M. Saeidi, W. Karwowski, F. V. Farahani, K. Fiok, R. Taiar, P. A. Hancock, and A. Al-Juaid, “Neural decoding of eeg signals with machine learning: a systematic review, ” Brain Sciences, vol. 11, no. 11, pp. 1525, 2021.
S. Biswas and R. Sinha, “Wavelet filterbank-based eeg rhythm-specific spatial features for covert speech classification, ” IET Signal Processing, vol. 16, no. 1, pp. 92-105, 2022.
T. Hernández-Del-Toro, C. A. Reyes-García, and L. Villaseñor-Pineda, “Toward asynchronous eeg-based bci: Detecting imagined words segments in continuous eeg signals, ” Biomedical Signal Processing and Control, vol. 65, pp. 102351, 2021.
M. A. Bakhshali, M. Khademi, A. Ebrahimi-Moghadam, and S. Moghimi, “Eeg signal classification of imagined speech based on riemannian distance of correntropy spectral density, ” Biomedical Signal Processing and Control, vol. 59, pp. 101899, 2020.
I. J. Moon and S. H. Hong, “What is temporal fine structure and why is it important?, ” Korean journal of audiology, vol. 18, no. 1, pp. 1, 2014.
G. Ni, Z. Xu, Y. Bai, Q. Zheng, R. Zhao, Y. Wu, and D. Ming, “Eeg-based assessment of temporal fine structure and envelope effect in mandarin syllable and tone perception, ” Cerebral Cortex, vol. 33, no. 23, pp. 11287-11299, 2023.
W. Zhang, M. Jiang, K. A. C. Teo, R. Bhuvanakantham, L. Fong, W. K. J. Sim, Z. Guo, C. H. V. Foo, R. H. J. Chua, P. Padmanabhan, et al., “Revealing the spatiotemporal brain dynamics of covert speech compared with overt speech: A simultaneous eeg-fmri study, ” NeuroImage, vol. 293, pp. 120629, 2024.
G. M. Oppenheim and G. S. Dell, “Motor movement matters: The flexible abstractness of inner speech, ” Memory & cognition, vol. 38, no. 8, pp. 1147-1160, 2010.
C. Cooney, R. Folli, and D. Coyle, “Optimizing layers improves cnn generalization and transfer learning for imagined speech decoding from eeg, ” in 2019 IEEE international conference on systems, man and cybernetics (SMC). IEEE, 2019, pp. 1311-1316.