[en] Biomedical multi-document summarization (BioMDSum) involves automatically generating concise and informative summaries from collections of related biomedical documents. While extractive summarization methods have shown promise, they often produce incoherent summaries. Onethe other hand, fully abstractive methods yield coherent summaries but demand extensive training datasets and computational resources due to the typically lengthy nature of biomedical documents. Toeaddress these challenges, weepropose a hybrid summarization method that combines the strengths of both approaches. The proposed method consists of two main phases: (i) an extractive summarization phase that uses k-means clustering to group similar sentences based on their cosine similarity between embeddings generated by the sentence-BERT model, followed by the PageRank algorithm for sentence scoring and selection; and (ii) an abstractive summarization phase that fine-Tunes a Longform Encoder-Decoder (LED) transformer model to generate a concise and coherent summary from the sentences selected during the extractive phase. Weeconducted several experiments on the standard biomedical multi-document summarization datasets Cochrane and MS2. The results demonstrate that the proposed method is competitive and outperforms recent state-of-The-Art systems based on ROUGE evaluation measures. Specifically, our model achieved ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and METEOR scores of 29.41%, 6.57%, 18.31%, 85.95%, and 22.15% on the Cochrane dataset, and 28.79%, 8.22%, 17.93%, 85.51%, and 25.17% on the MS 2 dataset, respectively. Furthermore, aneablation analysis shows that integrating extractive and abstractive phases in our hybrid summarization method enhances the overall performance of the proposed approach.
Disciplines :
Sciences informatiques
Auteur, co-auteur :
Aftiss, Azzedine ; Ibn Tofail University, Engineering Sciences Laboratory, National School of Applied Sciences, Kenitra, Morocco
LAMSIYAH, Salima ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)
Ouatik El Alaoui, Said ; Ibn Tofail University, Engineering Sciences Laboratory, National School of Applied Sciences, Kenitra, Morocco
SCHOMMER, Christoph ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
BioMDSum: An Effective Hybrid Biomedical Multi-Document Summarization Method Based on PageRank and Longformer Encoder-Decoder
Date de publication/diffusion :
2024
Titre du périodique :
IEEE Access
ISSN :
2169-3536
Maison d'édition :
Institute of Electrical and Electronics Engineers Inc.
A. Chaves, C. Kesiku, and B. Garcia-Zapirain, ‘‘Automatic text summarization of biomedical text data: A systematic review,’’ Information, vol. 13, no. 8, p. 393, Aug. 2022.
M. Nasr Azadani, N. Ghadiri, and E. Davoodijam, ‘‘Graph-based biomedical text summarization: An itemset mining and sentence clustering approach,’’ J. Biomed. Informat., vol. 84, pp. 42–58, Aug. 2018.
A. P. Widyassari, S. Rustad, G. F. Shidik, E. Noersasongko, A. Syukur, A. Affandy, and D. R. I. M. Setiadi, ‘‘Review of automatic text summarization techniques & methods,’’ J. King Saud Univ.-Comput. Inf. Sci., vol. 34, no. 4, pp. 1029–1046, 2022.
I. Bartolini, V. Moscato, M. Postiglione, G. Sperlì, and A. Vignali, ‘‘Data augmentation via context similarity: An application to biomedical named entity recognition,’’ Inf. Syst., vol. 119, Oct. 2023, Art. no. 102291.
H. Alami, A. El Mahdaouy, A. Benlahbib, N. En-Nahnahi, I. Berrada, and S. E. A. Ouatik, ‘‘DAQAS: Deep Arabic question answering system based on duplicate question detection and machine reading comprehension,’’ J. King Saud Univ.-Comput. Inf. Sci., vol. 35, no. 8, Sep. 2023, Art. no. 101709.
M. K. Rohil and V. Magotra, ‘‘An exploratory study of automatic text summarization in biomedical and healthcare domain,’’ Healthcare Anal., vol. 2, Nov. 2022, Art. no. 100058.
M. Allahyari, S. Pouriyeh, M. Assefi, S. Safaei, E. D. Trippe, J. B. Gutierrez, and K. Kochut, ‘‘Text summarization techniques: A brief survey,’’ 2017, arXiv:1707.02268.
S. Lamsiyah, A. E. Mahdaouy, and C. Schommer, ‘‘Can anaphora resolution improve extractive query-focused multi-document summarization?’’ IEEE Access, vol. 11, pp. 99961–99976, 2023.
Z. Zhang, X. Liang, Y. Zuo, and Z. Li, ‘‘Unsupervised abstractive summarization via sentence rewriting,’’ Comput. Speech Lang., vol. 78, Mar. 2023, Art. no. 101467.
W. S. El-Kassas, C. R. Salama, A. A. Rafea, and H. K. Mohamed, ‘‘Automatic text summarization: A comprehensive survey,’’ Expert Syst. Appl., vol. 165, Mar. 2021, Art. no. 113679.
X. Wan, ‘‘Towards a unified approach to simultaneous single-document and multi-document summarizations,’’ in Proc. 23rd Int. Conf. Comput. Linguistics (Coling), Aug. 2010, pp. 1137–1145.
N. Reimers and I. Gurevych, ‘‘Sentence-BERT: Sentence embeddings using Siamese BERT-networks,’’ 2019, arXiv:1908.10084.
L. Page, S. Brin, R. Motwani, and T. Winograd, ‘‘The PageRank citation ranking: Bringing order to the web,’’ Stanford InfoLab, Stanford, CA, USA, Tech. Rep. 1999-66, 1999.
I. Beltagy, M. E. Peters, and A. Cohan, ‘‘Longformer: The long-document transformer,’’ 2020, arXiv:2004.05150.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, ‘‘Exploring the limits of transfer learning with a unified text-to-text transformer,’’ J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, Jan. 2020.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ‘‘BERT: Pre-training of deep bidirectional transformers for language understanding,’’ 2018, arXiv:1810.04805.
J. Zhang, Y. Zhao, M. Saleh, and P. Liu, ‘‘PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization,’’ in Proc. Int. Conf. Mach. Learn., Jul. 2020, pp. 11328–11339.
M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, ‘‘Big bird: Transformers for longer sequences,’’ in Proc. Adv. Neural Inf. Process. Syst., vol. 33, Jan. 2020, pp. 17283–17297.
M. Guo, J. Ainslie, D. Uthus, S. Ontanon, J. Ni, Y.-H. Sung, and Y. Yang, ‘‘LongT5: Efficient text-to-text transformer for long sequences,’’ 2021, arXiv:2112.07916.
D. Alahmadi, A. Wali, and S. Alzahrani, ‘‘TAAM: Topic-aware abstractive Arabic text summarisation using deep recurrent neural networks,’’ J. King Saud Univ.-Comput. Inf. Sci., vol. 34, no. 6, pp. 2651–2665, Jun. 2022.
Á. Díaz-Pacheco, R. Guerrero-Rodríguez, M. Á. Álvarez-Carmona, A. Y. Rodríguez-González, and R. Aranda, ‘‘A comprehensive deep learning approach for topic discovering and sentiment analysis of textual information in tourism,’’ J. King Saud Univ.-Comput. Inf. Sci., vol. 35, no. 9, Oct. 2023, Art. no. 101746.
A. Paramasivam and S. J. Nirmala, ‘‘A survey on textual entailment based question answering,’’ J. King Saud Univ.-Comput. Inf. Sci., vol. 34, no. 10, pp. 9644–9653, Nov. 2022.
A. P. Bhopale and A. Tiwari, ‘‘Transformer based contextual text representation framework for intelligent information retrieval,’’ Expert Syst. Appl., vol. 238, Mar. 2024, Art. no. 121629.
F.-Z. El-Alami, S. O. E. Alaoui, and N. E. Nahnahi, ‘‘Contextual semantic embeddings based on fine-tuned AraBERT model for Arabic text multiclass categorization,’’ J. King Saud Univ.-Comput. Inf. Sci., vol. 34, no. 10, pp. 8422–8428, Nov. 2022.
Q. Xie, Z. Luo, B. Wang, and S. Ananiadou, ‘‘A survey for biomedical text summarization: From pre-trained to large language models,’’ 2023, arXiv:2304.08763.
M. Moradi, M. Dashti, and M. Samwald, ‘‘Summarization of biomedical articles using domain-specific word embeddings and graph ranking,’’ J. Biomed. Informat., vol. 107, Jul. 2020, Art. no. 103452.
E. Davoodijam, N. Ghadiri, M. Lotfi Shahreza, and F. Rinaldi, ‘‘MultiGBS: A multi-layer graph approach to biomedical summarization,’’ J. Biomed. Informat., vol. 116, Apr. 2021, Art. no. 103706.
M. Moradi and N. Ghadiri, ‘‘Different approaches for identifying important concepts in probabilistic biomedical text summarization,’’ Artif. Intell. Med., vol. 84, pp. 101–116, Jan. 2018.
O. Rouane, H. Belhadef, and M. Bouakkaz, ‘‘Combine clustering and frequent itemsets mining to enhance biomedical text summarization,’’ Expert Syst. Appl., vol. 135, pp. 362–373, Nov. 2019.
S. Lamsiyah, A. E. Mahdaouy, B. Espinasse, and S. E. A. Ouatik, ‘‘An unsupervised method for extractive multi-document summarization based on centroid approach and sentence embeddings,’’ Expert Syst. Appl., vol. 167, Apr. 2021, Art. no. 114152.
Y. Du, Q. Li, L. Wang, and Y. He, ‘‘Biomedical-domain pre-trained language model for extractive summarization,’’ Knowl.-Based Syst., vol. 199, Jul. 2020, Art. no. 105964.
Q. Xie, J. A. Bishop, P. Tiwari, and S. Ananiadou, ‘‘Pre-trained language models with domain knowledge for biomedical extractive summarization,’’ Knowl.-Based Syst., vol. 252, Sep. 2022, Art. no. 109460.
A. See, P. J. Liu, and C. D. Manning, ‘‘Get to the point: Summarization with pointer-generator networks,’’ 2017, arXiv:1704.04368.
A. Alambo, T. Banerjee, K. Thirunarayan, and M. Raymer, ‘‘Entity-driven fact-aware abstractive summarization of biomedical literature,’’ in Proc. 26th Int. Conf. Pattern Recognit. (ICPR), Aug. 2022, pp. 613–620.
P. Gigioli, N. Sagar, J. Voyles, and A. Rao, ‘‘Domain-aware abstractive text summarization for medical documents,’’ in Proc. IEEE Int. Conf. Bioinf. Biomed. (BIBM), Dec. 2018, pp. 2338–2343.
A. Soleimani, V. Nikoulina, B. Favre, and S. Ait Mokhtar, ‘‘Zero-shot aspect-based scientific document summarization using self-supervised pre-training,’’ in Proc. 21st Workshop Biomed. Lang. Process., 2022, pp. 49–62.
E. Lalitha, K. Ramani, D. Shahida, E. V. S. Deepak, M. H. Bindu, and D. Shaikshavali, ‘‘Text summarization of medical documents using abstractive techniques,’’ in Proc. 2nd Int. Conf. Appl. Artif. Intell. Comput. (ICAAIC), May 2023, pp. 939–943.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, ‘‘BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,’’ 2019, arXiv:1910.13461.
C. Tang, S. Wang, T. Goldsack, and C. Lin, ‘‘Improving biomedical abstractive summarisation with knowledge aggregation from citation papers,’’ 2023, arXiv:2310.15684.
B. C. Wallace, S. Saha, F. Soboczenski, and I. J. Marshall, ‘‘Generating (factual?) narrative summaries of RCTs: Experiments with neural multidocument summarization,’’ AMIA Summits Transl. Sci. Proc., vol. 2021, pp. 605–614, Jan. 2021.
Y. Guo, W. Qiu, Y. Wang, and T. Cohen, ‘‘Automated lay language summarization of biomedical scientific reviews,’’ in Proc. AAAI Conf. Artif. Intell., vol. 35, May 2021, pp. 160–168.
Y. Otmakhova, H. T. Truong, T. Baldwin, T. Cohn, K. Verspoor, and J. H. Lau, ‘‘LED down the rabbit hole: Exploring the potential of global attention for biomedical multi-document summarisation,’’ 2022, arXiv:2209.08698.
W. Xiao, I. Beltagy, G. Carenini, and A. Cohan, ‘‘PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization,’’ 2021, arXiv:2110.08499.
R. Tangsali, A. J. Vyawahare, A. V. Mandke, O. R. Litake, and D. D. Kadam, ‘‘Abstractive approaches to multidocument summarization of medical literature reviews,’’ in Proc. 3rd Workshop Scholarly Document Process., 2022, pp. 199–203.
B. Yu, ‘‘Evaluating pre-trained language models on multi-document summarization for literature reviews,’’ in Proc. 3rd Workshop Scholarly Document Process., 2022, pp. 188–192.
S. Shleifer and A. M. Rush, ‘‘Pre-trained summarization distillation,’’ 2020, arXiv:2010.13002.
G. Sharma and D. Sharma, ‘‘Automatic text summarization methods: A comprehensive review,’’ Social Netw. Comput. Sci., vol. 4, no. 1, p. 33, Oct. 2022.
W.-T. Hsu, C.-K. Lin, M.-Y. Lee, K. Min, J. Tang, and M. Sun, ‘‘A unified model for extractive and abstractive summarization using inconsistency loss,’’ 2018, arXiv:1805.06266.
P. Muniraj, K. R. Sabarmathi, R. Leelavathi, and B. S. Balaji, ‘‘HNTSumm: Hybrid text summarization of transliterated news articles,’’ Int. J. Intell. Netw., vol. 4, pp. 53–61, Jan. 2023.
R. Mihalcea and P. Tarau, ‘‘TextRank: Bringing order into text,’’ in Proc. Conf. Empirical Methods Natural Lang. Process., Jul. 2004, pp. 404–411.
A. Gidiotis and G. Tsoumakas, ‘‘A divide-and-conquer approach to the summarization of long documents,’’ IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 28, pp. 3029–3040, 2020.
J. Bishop, Q. Xie, and S. Ananiadou, ‘‘GenCompareSum: A hybrid unsupervised summarization method using salience,’’ in Proc. 21st Workshop Biomed. Lang. Process., 2022, pp. 220–240.
A. Ghadimi and H. Beigy, ‘‘Hybrid multi-document summarization using pre-trained language models,’’ Expert Syst. Appl., vol. 192, Apr. 2022, Art. no. 116292.
H. Li, A. Einolghozati, S. Iyer, B. Paranjape, Y. Mehdad, S. Gupta, and M. Ghazvininejad, ‘‘EASE: Extractive-abstractive summarization end-to-end using the information bottleneck principle,’’ in Proc. 3rd Workshop New Frontiers Summarization, 2021, pp. 85–95.
Y. Du, Y. Zhao, J. Yan, and Q. Li, ‘‘UGDAS: Unsupervised graph-network based denoiser for abstractive summarization in biomedical domain,’’ Methods, vol. 203, pp. 160–166, Jul. 2022.
J. L. D. Nunna, V. Hanuman Turaga, and S. Chebrolu, ‘‘Extractive and abstractive text summarization model fine-tuned based on BERTSUM and bio-BERT on COVID-19 open research articles,’’ in Proc. Int. Conf. Mach. Learn. Big Data Anal. Cham, Switzerland: Springer, 2022, pp. 213–223.
Y. Liu and M. Lapata, ‘‘Text summarization with pretrained encoders,’’ 2019, arXiv:1908.08345.
A. Givchi, R. Ramezani, and A. Baraani-Dastjerdi, ‘‘Graph-based abstractive biomedical text summarization,’’ J. Biomed. Informat., vol. 132, Aug. 2022, Art. no. 104099.
K. Shinde, T. Roy, and T. Ghosal, ‘‘An extractive-abstractive approach for multi-document summarization of scientific articles for literature review,’’ in Proc. 3rd Workshop Scholarly Document Process., 2022, pp. 204–209.
D. Miller, ‘‘Leveraging BERT for extractive text summarization on lectures,’’ 2019, arXiv:1906.04165.
J. Giorgi, L. Soldaini, B. Wang, G. Bader, K. Lo, L. Lu Wang, and A. Cohan, ‘‘Open domain multi-document summarization: A comprehensive study of model brittleness under retrieval,’’ 2022, arXiv:2212.10526.
G. Qin, Y. Feng, and B. Van Durme, ‘‘The NLP task effectiveness of long-range transformers,’’ 2022, arXiv:2202.07856.
T. Rehman, R. Mandal, A. Agarwal, and D. K. Sanyal, ‘‘Hallucination reduction in long input text summarization,’’ 2023, arXiv:2309.16781.
U. Khanna, S. Ghodratnama, and A. Beheshti, ‘‘Transformer-based models for long document summarisation in financial domain,’’ in Proc. 4th Financial Narrative Process. Workshop, Jun. 2022, pp. 73–78.
Y. Lan, Y. Wu, W. Xu, W. Feng, and Y. Zhang, ‘‘Chinese fine-grained financial sentiment analysis with large language models,’’ 2023, arXiv:2306.14096.
R. Kimera, D. N. Rim, J. Kirabira, U. G. Udomah, and H. Choi, ‘‘Enhanced labeling technique for reddit text and fine-tuned Longformer models for classifying depression severity in English and Luganda,’’ in Proc. 14th Int. Conf. Inf. Commun. Technol. Converg. (ICTC), Oct. 2023, pp. 76–80.
C. Xiao, X. Hu, Z. Liu, C. Tu, and M. Sun, ‘‘Lawformer: A pre-trained language model for Chinese legal long documents,’’ AI Open, vol. 2, pp. 79–84, May 2021.
J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah, ‘‘Signature verification using a ‘Siamese’ time delay neural network,’’ Int. J. Pattern Recognit. Artif. Intell., vol. 7, no. 4, pp. 669–688, Aug. 1993.
S. Lamsiyah and C. Schommer, ‘‘A comparative study of sentence embeddings for unsupervised extractive multi-document summarization,’’ in Proc. Artif. Intell. Mach. Learn., T. Calders, C. Vens, J. Lijffijt, and B. Goethals, Eds. Cham, Switzerland: Springer, Jan. 2023, pp. 78–95.
K. Krishna and M. N. Murty, ‘‘Genetic K-means algorithm,’’ IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 29, no. 3, pp. 433–439, Jun. 1999.
J. DeYoung, I. Beltagy, M. van Zuylen, B. Kuehl, and L. L. Wang, ‘‘MS2: Multi-document summarization of medical studies,’’ 2021, arXiv:2104.06486.
C.-Y. Lin, ‘‘ROUGE: A package for automatic evaluation of summaries,’’ in Proc. Text summarization branches out, Jul. 2004, pp. 74–81.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, ‘‘BERTScore: Evaluating text generation with BERT,’’ 2019, arXiv:1904.09675.
S. Banerjee and A. Lavie, ‘‘METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,’’ in Proc. ACL Workshop Intrinsic Extrinsic Eval. Measures Mach. Transl. Summarization, Jun. 2005, pp. 65–72.
J. Steinberger and K. Jezek, ‘‘Using latent semantic analysis in text summarization and summary evaluation,’’ in Proc. ISIM, 2004, vol. 4, nos. 93–100, p. 8.
A. Haghighi and L. Vanderwende, ‘‘Exploring content models for multi-document summarization,’’ in Proc. Hum. Lang. Technol., Annu. Conf. North Amer. Chapter Assoc. Comput. Linguistics (NAACL), 2009, pp. 362–370.
H. P. Luhn, ‘‘The automatic creation of literature abstracts,’’ IBM J. Res. Develop., vol. 2, no. 2, pp. 159–165, Apr. 1958.
G. Erkan and D. R. Radev, ‘‘LexRank: Graph-based lexical centrality as salience in text summarization,’’ J. Artif. Intell. Res., vol. 22, pp. 457–479, Dec. 2004.
K. Huang, J. Altosaar, and R. Ranganath, ‘‘ClinicalBERT: Modeling clinical notes and predicting hospital readmission,’’ 2019, arXiv:1904.05342.
Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, ‘‘Domain-specific language model pretraining for biomedical natural language processing,’’ ACM Trans. Comput. Healthcare, vol. 3, no. 1, pp. 1–23, Jan. 2022.
S. Rothe, S. Narayan, and A. Severyn, ‘‘Leveraging pre-trained checkpoints for sequence generation tasks,’’ Trans. Assoc. Comput. Linguistics, vol. 8, pp. 264–280, Dec. 2020.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, ‘‘RoBERTa: A robustly optimized BERT pretraining approach,’’ 2019, arXiv:1907.11692.
H. Touvron et al., ‘‘Llama 2: Open foundation and fine-tuned chat models,’’ 2023, arXiv:2307.09288.
M. Das, K. Selvakumar, and P. J. A. Alphonse, ‘‘A comparative study on TF-IDF feature weighting method and its analysis using unstructured dataset,’’ 2023, arXiv:2308.04037.
J. Pennington, R. Socher, and C. Manning, ‘‘Glove: Global vectors for word representation,’’ in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2014, pp. 1532–1543.
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, ‘‘Enriching word vectors with subword information,’’ Trans. Assoc. Comput. Linguistics, vol. 5, pp. 135–146, Dec. 2017.
F. Murtagh and P. Contreras, ‘‘Algorithms for hierarchical clustering: An overview,’’ Wiley Interdiscipl. Rev., Data Mining Knowl. Discovery, vol. 2, no. 1, pp. 86–97, Jan. 2012.
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, ‘‘Density-based spatial clustering of applications with noise,’’ in Proc. Int. Conf. Knowl. Discovery Data Mining, vol. 240, 1996, pp. 226–231.
