Fine-Tuning a Large Language Model with Reinforcement Learning for Educational Question Generation

LAMSIYAH, Salima; El Mahdaouy, Abdelkader; NOURBAKHSH, Aria; SCHOMMER, Christoph

doi:10.1007/978-3-031-64302-6_30

No full text

Contribution to collective works (Parts of books)

Fine-Tuning a Large Language Model with Reinforcement Learning for Educational Question Generation

LAMSIYAH, Salima; El Mahdaouy, Abdelkader; NOURBAKHSH, Aria et al.

2024 • In Lecture Notes in Computer Science

Peer reviewed

Permalink
https://hdl.handle.net/10993/61779

DOI
10.1007/978-3-031-64302-6_30

Files (0)Send to Details Statistics Bibliography Similar publications

Files

Full Text

No document available.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Educational Question Generation · Large Language Model · Google FLAN-T5 · Reinforcement Learning · Self-Critical Sequence Training

Abstract :

[en] Educational Natural Language Generation (EduQG) aims to automatically generate educational questions from textual content, which is crucial for the expansion of online education. Prior research in EduQG has predominantly relied on cross-entropy loss for training, which can lead to issues such as exposure bias and inconsistencies between training and testing metrics. To mitigate this issue, we propose a reinforcement learning (RL) based large language model (LLM) for educational question generation. In particular, we fine-tune the Google FLAN-T5 model using a mixed objective function that combines cross-entropy and RL losses to ensure the generation of questions that are syntactically and semantically accurate. The experimental results on the SciQ question generation dataset show that the proposed method is competitive with current state-of-the-art systems in terms of predictive performance and linguistic quality.

Disciplines :

Computer science

Author, co-author :

LAMSIYAH, Salima ; University of Luxembourg

El Mahdaouy, Abdelkader

NOURBAKHSH, Aria ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

SCHOMMER, Christoph ; University of Luxembourg

External co-authors :

yes

Language :

English

Title :

Fine-Tuning a Large Language Model with Reinforcement Learning for Educational Question Generation

Publication date :

04 July 2024

Main work title :

Lecture Notes in Computer Science

Publisher :

Springer Nature Switzerland, recife, Brazil

ISBN/EAN :

978-3-03-164302-6
978-3-03-164301-9

Peer reviewed :

Peer reviewed

Focus Area :

Educational Sciences

Additional URL :

https://link.springer.com/content/pdf/10.1007/978-3-031-64302-6_30

Available on ORBilu :

since 04 August 2024

Statistics

Number of views

197 (20 by Unilu)

Number of downloads

0 (0 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Bulathwela, S., Muse, H., Yilmaz, E.: Scalable educational question generation with pre-trained language models. In: International Conference on Artificial Intelligence in Education, pp. 327–339. Springer (2023). https://doi.org/10.1007/978-3-031-36272-9 27
Chen, Y., Wu, L., Zaki, M.J.: Reinforcement learning based graph-to-sequence model for natural question generation. arXiv preprint arXiv:1908.04942 (2019)
Danon, G., Last, M.: A syntactic approach to domain-specific automatic question generation. arXiv preprint arXiv:1712.09827 (2017)
Das, B., Majumder, M., Phadikar, S., Sekh, A.A.: Automatic question generation and answer assessment: a survey. Res. Pract. Technol. Enhanc. Learn. 16(1), 1–15 (2021)
Du, X., Shao, J., Cardie, C.: Learning to ask: neural question generation for reading comprehension. In: Barzilay, R., Kan, M.Y. (eds.) Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1342–1352 (2017)
Elkins, S., Kochmar, E., Serban, I., Cheung, J.C.: How useful are educational questions generated by large language models? In: International Conference on Artificial Intelligence in Education, pp. 536–542. Springer (2023). https://doi.org/10.1007/978-3-031-36336-8_83
Gao, T., Yao, X., Chen, D.: SimCSE: simple contrastive learning of sentence embeddings. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894–6910 (2021)
Gou, Q., et al.: Diversify question generation with retrieval-augmented style transfer. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1677–1690. Association for Computational Linguistics (2023)
Guan, M., Mondal, S.K., Dai, H.N., Bao, H.: Reinforcement learning-driven deep question generation with rich semantics. Inf. Process. Manage. 60(2), 103232 (2023)
Heilman, M., Smith, N.A.: Good question! statistical ranking for question generation. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 609–617 (2010)
Jiao, Y., Shridhar, K., Cui, P., Zhou, W., Sachan, M.: Automatic educational question generation with difficulty level controls. In: International Conference on Artificial Intelligence in Education, pp. 476–488. Springer (2023). https://doi.org/10.1007/978-3-031-36272-9 39
Keneshloo, Y., Shi, T., Ramakrishnan, N., Reddy, C.K.: Deep reinforcement learning for sequence-to-sequence models. IEEE Trans. Neural Netw. Learn. Syst. 31(7), 2469–2489 (2019)
Kumar, V., Ramakrishnan, G., Li, Y.F.: Putting the Horse Before the Cart: a generator-evaluator framework for question generation from text. In: Bansal, M., Villavicencio, A. (eds.) Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 812–821 (2019)
Lamsiyah, S., Schommer, C.: A comparative study of sentence embeddings for unsupervised extractive multi-document summarization. In: Benelux Conference on Artificial Intelligence, pp. 78–95. Springer (2022). https://doi.org/10.1007/978-3-031-39144-6 6
Leite, B., Cardoso, H.L.: Towards enriched controllability for educational question generation. In: International Conference on Artificial Intelligence in Education, pp. 786–791. Springer (2023). https://doi.org/10.1007/978-3-031-36272-9 72
Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
Lo, K., Wang, L.L., Neumann, M., Kinney, R., Weld, D.: S2ORC: the semantic scholar open research corpus. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983 (2020)
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., Vanderwende, L.: Generating natural questions about an image. arXiv preprint arXiv:1603.06059 (2016)
Naeiji, A., An, A., Davoudi, H., Delpisheh, M., Alzghool, M.: Question generation using sequence-to-sequence model with semantic role labels. In: Vlachos, A., Augenstein, I. (eds.) Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2830–2842 (2023)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Isabelle, P., Charniak, E., Lin, D. (eds.) Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)
Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 (2017)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1) (2020)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392 (2016)
Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7008–7024 (2017)
Rus, V., Cai, Z., Graesser, A.: Question generation: example of a multi-year evaluation campaign. In: Proceedings of WS on the QGSTEC (2008)
Serban, I.V., et al.: Generating factoid questions with recurrent neural networks: the 30M factoid question-answer corpus. In: Erk, K., Smith, N.A. (eds.) Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 588–598 (2016)
Shimmei, M., Bier, N., Matsuda, N.: Machine-generated questions attract instructors when acquainted with learning objectives. In: International Conference on Artificial Intelligence in Education, pp. 3–15. Springer (2023). https://doi.org/10. 1007/978-3-031-36272-9 1
Song, L., Wang, Z., Hamza, W.: A unified query-based generative model for question generation and question answering. arXiv preprint arXiv:1709.01058 (2017)
Song, L., Wang, Z., Hamza, W., Zhang, Y., Gildea, D.: Leveraging context information for natural question generation. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 2 (Short Papers), pp. 569–574 (2018)
Wang, L., Xu, Z., Lin, Z., Zheng, H., Shen, Y.: Answer-driven deep question generation based on reinforcement learning. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 5159–5170 (2020)
Welbl, J., Liu, N.F., Gardner, M.: Crowdsourcing multiple choice science questions. In: Derczynski, L., Xu, W., Ritter, A., Baldwin, T. (eds.) Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94–106 (2017)
Zhang, Z.: Improved Adam optimizer for deep neural networks. In: 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pp. 1– 2. IEEE (2018)
Zhao, Z., Hou, Y., Wang, D., Yu, M., Liu, C., Ma, X.: Educational question generation of children storybooks via question type distribution learning and event-centric summarization. In: Muresan, S., Nakov, P., Villavicencio, A. (eds.) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5073–5085 (2022)
Zhou, Q., Yang, N., Wei, F., Tan, C., Bao, H., Zhou, M.: Neural question generation from text: a preliminary study. In: Huang, X., Jiang, J., Zhao, D., Feng, Y., Hong, Yu. (eds.) NLPCC 2017. LNCS (LNAI), vol. 10619, pp. 662–671. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73618-1_56