Clinical natural language processing; autoencoder; cardiac failure; sparsity; Humans; Neural Networks, Computer; Correlation of Data; Algorithms; Data Compression; Auto encoders; Features extraction; Language processing; Natural languages; Neural-networks; Sparse matrices; Task analysis; X-ray imaging; Biomedical Engineering
Résumé :
[en] [en] UNLABELLED: When dealing with clinical text classification on a small dataset, recent studies have confirmed that a well-tuned multilayer perceptron outperforms other generative classifiers, including deep learning ones. To increase the performance of the neural network classifier, feature selection for the learning representation can effectively be used. However, most feature selection methods only estimate the degree of linear dependency between variables and select the best features based on univariate statistical tests. Furthermore, the sparsity of the feature space involved in the learning representation is ignored.
GOAL: Our aim is, therefore, to access an alternative approach to tackle the sparsity by compressing the clinical representation feature space, where limited French clinical notes can also be dealt with effectively.
METHODS: This study proposed an autoencoder learning algorithm to take advantage of sparsity reduction in clinical note representation. The motivation was to determine how to compress sparse, high-dimensional data by reducing the dimension of the clinical note representation feature space. The classification performance of the classifiers was then evaluated in the trained and compressed feature space.
RESULTS: The proposed approach provided overall performance gains of up to 3% for each test set evaluation. Finally, the classifier achieved 92% accuracy, 91% recall, 91% precision, and 91% f1-score in detecting the patient's condition. Furthermore, the compression working mechanism and the autoencoder prediction process were demonstrated by applying the theoretic information bottleneck framework. Clinical and Translational Impact Statement- An autoencoder learning algorithm effectively tackles the problem of sparsity in the representation feature space from a small clinical narrative dataset. Significantly, it can learn the best representation of the training data because of its lossless compression capacity compared to other approaches. Consequently, its downstream classification ability can be significantly improved, which cannot be done using deep learning models.
Disciplines :
Ingénierie électrique & électronique
Auteur, co-auteur :
LE, Thanh-Dung ; University of Luxembourg ; Biomedical Information Processing Laboratory, Ecole de Technologie SuperieureUniversity of Quebec Montreal QC H3C 1K3 Canada ; Research Center at CHU Sainte-JustineUniversity of Montreal Montreal QC H3T 1J4 Canada
Noumeir, Rita ; Biomedical Information Processing Laboratory, Ecole de Technologie SuperieureUniversity of Quebec Montreal QC H3C 1K3 Canada
Rambaud, Jerome; Research Center at CHU Sainte-JustineUniversity of Montreal Montreal QC H3T 1J4 Canada
Sans, Guillaume; Research Center at CHU Sainte-JustineUniversity of Montreal Montreal QC H3T 1J4 Canada
Jouvet, Philippe ; Research Center at CHU Sainte-JustineUniversity of Montreal Montreal QC H3T 1J4 Canada
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learning.
Date de publication/diffusion :
2023
Titre du périodique :
IEEE Journal of Translational Engineering in Health and Medicine
ISSN :
2168-2372
Maison d'édition :
Institute of Electrical and Electronics Engineers Inc., Etats-Unis
Natural Sciences and Engineering Research Council Institut de Valorisation des Donnees de l’Universite de Montreal Fonds de la Recherche en Sante du Quebec Fonds de Recherche du Quebec-Nature et Technologies
Subventionnement (détails) :
This work was supported in part by the Natural Sciences and Engineering Research Council (NSERC).
M. A. Musen, B. Middleton, and R. A. Greenes, "Clinical decision-support systems," in Biomedical Informatics. Cham, Switzerland: Springer, 2021, pp. 795-840.
R. T. Sutton, D. Pincock, D. C. Baumgart, D. C. Sadowski, R. N. Fedorak, and K. I. Kroeker, "An overview of clinical decision support systems: Benefits, risks, and strategies for success," NPJ Digit. Med., vol. 3, no. 1, pp. 1-10, Feb. 2020.
R. Gold et al., "Effect of clinical decision support at community health centers on the risk of cardiovascular disease: A cluster randomized clinical trial," JAMA Netw. Open, vol. 5, no. 2, 2022, Art. no. e2146519.
G. Bellani et al., "Epidemiology, patterns of care, and mortality for patients with acute respiratory distress syndrome in intensive care units in 50 countries," JAMA, vol. 315, no. 8, pp. 788-800, 2016.
P. Jouvet, "Pediatric acute respiratory distress syndrome: Consensus recommendations from the pediatric acute lung injury consensus conference," Pediatric Crit. Care Med., J. Soc. Crit. Med.World Fed. Pediatric Intensive Crit. Care Societies, vol. 16, no. 5, p. 428, 2015.
M. Sauthier, G. Tuli, P. A. Jouvet, J. S. Brownstein, and A. G. Randolph, "Estimated Pao2: A continuous and noninvasive method to estimate Pao2 and oxygenation index," Crit. Care Explor., vol. 3, no. 10, p. e0546, 2021.
N. Zaglam, P. Jouvet, O. Flechelles, G. Emeriaud, and F. Cheriet, "Computer-aided diagnosis system for the acute respiratory distress syndrome from chest radiographs," Comput. Biol. Med., vol. 52, pp. 41-48, Sep. 2014.
M. Yahyatabar, P. Jouvet, and F. Cheriet, "Dense-Unet: A light model for lung fields segmentation in chest X-ray images," in Proc. 42nd Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC), Jul. 2020, pp. 1242-1245.
T.-D. Le, R. Noumeir, J. Rambaud, G. Sans, and P. Jouvet, "Detecting of a patient's condition from clinical narratives using natural language representation," IEEE Open J. Eng. Med. Biol., vol. 3, pp. 142-149, 2022.
T.-D. Le, R. Noumeir, J. Rambaud, G. Sans, and P. Jouve, "Machine learning based on natural language processing to detect cardiac failure in clinical narratives," in Proc. 36th Congr. Recherche, 2021, pp. 1-6.
D. Jain and V. Singh, "Feature selection and classification systems for chronic disease prediction: A review," Egyptian Informat. J., vol. 19, no. 3, pp. 179-189, Nov. 2018.
G. Forman, "An extensive empirical study of feature selection metrics for text classification," J. Mach. Learn. Res., vol. 3, pp. 1289-1305, Mar. 2003.
C. Zhou, Y. Jia, and M. Motani, "Optimizing autoencoders for learning deep representations from health data," IEEE J. Biomed. Health Informat., vol. 23, no. 1, pp. 103-111, Jan. 2019.
Y. Xiong and Y. Lu, "Deep feature extraction from the vocal vectors using sparse autoencoders for Parkinson's classification," IEEE Access, vol. 8, pp. 27821-27830, 2020.
P. Kolyvakis, A. Kalousis, B. Smith, and D. Kiritsis, "Biomedical ontology alignment: An approach based on representation learning," J. Biomed. Semantics, vol. 9, no. 1, pp. 1-20, Dec. 2018.
J. C. Quiroz, L. Laranjo, A. B. Kocaballi, S. Berkovsky, D. Rezazadegan, and E. Coiera, "Challenges of developing a digital scribe to reduce clinical documentation burden," NPJ Digit. Med., vol. 2, no. 1, p. 114, Nov. 2019.
M. L. Abadi, L. Labiod, and M. Nadif, "Denoising autoencoder as an effective dimensionality reduction and clustering of text data," in Proc. Pacific-Asia Conf. Knowl. Discovery Data Mining, 2017, pp. 801-813.
N. Hurley and S. Rickard, "Comparing measures of sparsity," IEEE Trans. Inf. Theory, vol. 55, no. 10, pp. 4723-4741, Oct. 2009.
M. A. Kramer, "Nonlinear principal component analysis using autoassociative neural networks," AIChE J., vol. 37, no. 2, pp. 233-243, Feb. 1991.
G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
Y. Wang, H. Yao, and S. Zhao, "Auto-encoder based dimensionality reduction," Neurocomputing, vol. 184, pp. 232-242, Apr. 2016.
S. Garg and Y. Liang, "Functional regularization for representation learning: A unified theoretical perspective," in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 1-13.
C. Steinmeyer and L.Wiese, "Sampling methods and feature selection for mortality prediction with neural networks," J. Biomed. Informat., vol. 111, Nov. 2020, Art. no. 103580.
Y. Shi, M. Lei, R. Ma, and L. Niu, "Learning robust auto-encoders with regularizer for linearity and sparsity," IEEE Access, vol. 7, pp. 17195-17206, 2019.
S. Yu and J. C. Príncipe, "Understanding autoencoders with information theoretic concepts," Neural Netw., vol. 117, pp. 104-123, Sep. 2019.
N. I. Tapia and P. A. Estevez, "On the information plane of autoencoders," in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2020, pp. 1-8.
S. Lee and J. Jo, "Information flows of diverse autoencoders," Entropy, vol. 23, no. 7, p. 862, Jul. 2021.
N. Tishby, F. C. Pereira, and W. Bialek, "The information bottleneck method," 2000, arXiv:physics/0004057.
R. Shwartz-Ziv and N. Tishby, "Opening the black box of deep neural networks via information," 2017, arXiv:1703.00810.
B. C. Geiger, "On information plane analyses of neural network classifiers-A review," IEEE Trans. Neural Netw. Learn. Syst., vol. 33, no. 12, pp. 7039-7051, Dec. 2022.
B. C. Geiger and G. Kubin, "Information bottleneck: Theory and applications in deep learning," Entropy, vol. 22, no. 12, p. 1408, Dec. 2020.
M. A. Alomrani, "A critical review of information bottleneck theory and its applications to deep learning," 2021, arXiv:2105.04405.
C. Goutte and E. Gaussier, "A probabilistic interpretation of precision, recall and F-score, with implication for evaluation," in Proc. Eur. Conf. Inf. Retr. Cham, Switzerland: Springer, 2005, pp. 345-359.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," J. Mach. Learn. Res., vol. 15, no. 56, pp. 1929-1958, 2014.
X. Glorot and Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," in Proc. 13th Int. Conf. Artif. Intell. Statist., 2010, pp. 249-256.
Y. Lu, Y.-M. Cheung, and Y. Y. Tang, "Bayes imbalance impact index: A measure of class imbalanced data set for classification problem," IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 9, pp. 3525-3539, Sep. 2020.
F. Pedregosa et al., "Scikit-learn: Machine learning in Python," J. Mach. Learn. Res., vol. 12, pp. 2825-2830, Oct. 2012.
F. Chollet et al. (2015). Keras. [Online]. Available: https://keras.io
G. Luo, "A review of automatic selection methods for machine learning algorithms and hyper-parameter values," Netw. Model. Anal. Health Informat. Bioinf., vol. 5, no. 1, pp. 1-16, Dec. 2016.
T. Yu and H. Zhu, "Hyper-parameter optimization: A review of algorithms and applications," 2020, arXiv:2003.05689.
F. Anowar, S. Sadaoui, and B. Selim, "Conceptual and empirical comparison of dimensionality reduction algorithms (PCA, KPCA, LDA, MDS, SVD, LLE, ISOMAP, LE, ICA, t-SNE)," Comput. Sci. Rev., vol. 40, Jan. 2021, Art. no. 100378.
G. T. Reddy et al., "Analysis of dimensionality reduction techniques on big data," IEEE Access, vol. 8, pp. 54776-54788, 2020.
A. M. Martinez and A. C. Kak, "PCA versus LDA," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 228-233, Feb. 2002.
A. K. Gárate-Escamila, A. Hajjam El Hassani, and E. Andrès, "Classification models for heart disease prediction using feature selection and PCA," Informat. Med. Unlocked, vol. 19, 2020, Art. no. 100330.
A. Tharwat, T. Gaber, A. Ibrahim, and A. E. Hassanien, "Linear discriminant analysis: A detailed tutorial," AI Commun., vol. 30, no. 2, pp. 169-190, 2017.
J. Ghosh and S. B. Shuvo, "Improving classification model's performance using linear discriminant analysis on linear data," in Proc. 10th Int. Conf. Comput., Commun. Netw. Technol. (ICCCNT), Jul. 2019, pp. 1-5.
R. Dzisevic and D. Sesok, "Text classification using different feature extraction approaches," in Proc. Open Conf. Electr., Electron. Inf. Sci. (eStream), Apr. 2019, pp. 1-4.
S. W. Kim and J. M. Gil, "Research paper classification systems based on TF-IDF and LDA schemes," Hum.-Centric Comput. Inf. Sci., vol. 9, no. 1, p. 30, 2019.
Q. Chen, L. Yao, and J. Yang, "Short text classification based on LDA topic model," in Proc. Int. Conf. Audio, Lang. Image Process. (ICALIP), Jul. 2016, pp. 749-753.
S. Fodeh, T. Li, H. Jarad, and B. Safdar, "Classification of patients with coronary microvascular dysfunction," IEEE/ACM Trans. Comput. Biol. Bioinf., vol. 17, no. 2, pp. 704-711, Apr. 2019.
S. Laghmati, B. Cherradi, A. Tmiri, O. Daanouni, and S. Hamida, "Classification of patients with breast cancer using neighbourhood component analysis and supervised machine learning techniques," in Proc. 3rd Int. Conf. Adv. Commun. Technol. Netw. (CommNet), Sep. 2020, pp. 1-6.
J. Goldberger and E. A. Hinton, "Neighbourhood components analysis," in Proc. NeurIPS, 2005, pp. 513-520.
J. Gehring, Y. Miao, F. Metze, and A.Waibel, "Extracting deep bottleneck features using stacked auto-encoders," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., May 2013, pp. 3377-3381.
I. D. Mienye, Y. Sun, and Z. Wang, "Improved sparse autoencoder based artificial neural network approach for prediction of heart disease," Informat. Med. Unlocked, vol. 18, Jan. 2020, Art. no. 100307.
S. Arlot and A. Celisse, "Asurvey of cross-validation procedures for model selection," Statist. Surv., vol. 4, pp. 40-79, Jun. 2010.
A. Y. Ng and M. I. Jordan, "On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes," in Proc. Adv. Neural Inf. Process. Syst. (NIPS), vol. 14, 2002, p. 841.
J.-H. Xue and D. M. Titterington, "Comment on 'on discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes,"' Neural Process. Lett., vol. 28, no. 3, pp. 169-187, Dec. 2008.
P. Viola and W. M. Wells, "Alignment by maximization of mutual information," Int. J. Comput. Vis., vol. 24, no. 2, pp. 137-154, Sep. 1997.
J. P.W. Pluim, J. B. A. Maintz, and M. A. Viergever, "Mutual-informationbased registration of medical images: A survey," IEEE Trans. Med. Imag., vol. 22, no. 8, pp. 986-1004, Aug. 2003.
C. Olsen, P. E. Meyer, and G. Bontempi, "On the impact of entropy estimation on transcriptional regulatory network inference based on mutual information," EURASIP J. Bioinf. Syst. Biol., vol. 2009, pp. 1-9, Jan. 2009.
C. Rudin, "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead," Nature Mach. Intell., vol. 1, no. 5, pp. 206-215, 2019.
J. A. Fries et al., "Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences," Nature Commun., vol. 10, no. 1, pp. 1-10, 2019.
