Natural Language Processing (NLP); Large Language Models (LLMs); Prompt Engineering; ChatGPT; Model Generation; Sequence Diagrams
Résumé :
[en] Complementing natural language (NL) requirements with graphical models can improve stakeholders' communication and provide directions for system design. However, creating models from requirements involves manual effort. The advent of generative large language models (LLMs), ChatGPT being a notable example, offers promising avenues for automated assistance in model generation. This paper investigates the capability of ChatGPT to generate a specific type of model, i.e., UML sequence diagrams, from NL requirements. We conduct a qualitative study in which we examine the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Observations from the analysis of the generated diagrams have systematically been captured through evaluation logs, and categorized through thematic analysis. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their completeness and correctness with respect to the specified requirements often present challenges. This issue is particularly pronounced in the presence of requirements smells, such as ambiguity and inconsistency. The insights derived from this study can influence the practical utilization of LLMs in the RE process, and open the door to novel RE-specific prompting strategies targeting effective model generation.
Centre de recherche :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Disciplines :
Sciences informatiques
Auteur, co-auteur :
Ferrari, Alessio; Consiglio Nazionale Delle Ricerche (CNR), Italy
ABUALHAIJA, Sallam ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Arora, Chetan; Monash University, Australia
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
Model Generation with LLMs: From Requirements to UML Sequence Diagrams
Date de publication/diffusion :
2024
Nom de la manifestation :
The 14th International Model-Driven Requirements Engineering (MoDRE) workshop
R. Jolak, M. Savary-Leblanc, M. Dalibor, A. Wortmann, R. Hebig, J. Vincur, I. Polasek, X. Le Pallec, S. Gérard, and M. R. Chaudron, "Software engineering whispers: The effect of textual vs. graphical software design descriptions on software design communication, " Empirical software engineering, vol. 25, pp. 4427-4471, 2020.
S. Wagner, D. M. Fernández, M. Felderer, A. Vetrò, M. Kalinowski, R. Wieringa, D. Pfahl, T. Conte, M.-T. Christiansson, D. Greer et al., "Status quo in requirements engineering: A theory and a global family of surveys, " ACM TOSEM, vol. 28, no. 2, pp. 1-48, 2019.
V. Ambriola and V. Gervasi, "On the systematic analysis of natural language requirements with CIRCE, " ASE, vol. 13, pp. 107-167, 2006.
C. Arora, M. Sabetzadeh, S. Nejati, and L. Briand, "An active learning approach for improving the accuracy of automated domain model extraction, " ACM TOSEM, vol. 28, no. 1, pp. 1-34, 2019.
J. Horkoff, F. B. Aydemir, E. Cardoso, T. Li, A. Maté, E. Paja, M. Salnitri, L. Piras, J. Mylopoulos, and P. Giorgini, "Goal-oriented requirements engineering: An extended systematic mapping study, " REJ, vol. 24, pp. 133-160, 2019.
T. Yue, L. C. Briand, and Y. Labiche, "aToucan: An automated framework to derive UML analysis models from use case models, " ACM TOSEM, vol. 24, no. 3, pp. 1-52, 2015.
M. Jahan, Z. S. H. Abad, and B. Far, "Generating sequence diagram from natural language requirements, " in REW'21. IEEE, 2021, pp. 39-48.
R. Saini, G. Mussbacher, J. L. Guo, and J. Kienzle, "Automated, interactive, and traceable domain modelling empowered by artificial intelligence, " SoSym, pp. 1-31, 2022.
C. Arora, J. Grundy, and M. Abdelrazek, "Advancing requirements engineering through generative AI: Assessing the role of LLMs, " arXiv preprint arXiv: 2310. 13976, 2023.
S. Ahmed, A. Ahmed, and N. U. Eisty, "Automatic transformation of natural to unified modeling language: A systematic review, " in SERA'22. IEEE, 2022, pp. 112-119.
B. Chen, K. Chen, S. Hassani, Y. Yang, D. Amyot, L. Lessard, G. Mussbacher, M. Sabetzadeh, and D. Varró, "On the use of GPT-4 for creating goal models: An exploratory study, " in REW'23. IEEE, 2023, pp. 262-271.
J. Cámara, J. Troya, L. Burgueño, and A. Vallecillo, "On the assessment of generative AI in modeling tasks: An experience report with ChatGPT and UML, " SoSym, pp. 1-13, 2023.
K. Chen, Y. Yang, B. Chen, J. A. H. López, G. Mussbacher, and D. Varró, "Automated domain modeling with large language models: A comparative study, " in MODELS'23. IEEE, 2023, pp. 162-172.
K.-J. Stol and B. Fitzgerald, "Guidelines for conducting software engineering research, " in Contemporary Empirical Methods in Software Engineering. Springer, 2020, pp. 27-62.
G. Guest, K. M. MacQueen, and E. E. Namey, Applied thematic analysis. sage publications, 2011.
L. Zhao, W. Alhoshan, A. Ferrari, K. J. Letsholo, M. A. Ajagbe, E.-V. Chioasca, and R. T. Batista-Navarro, "Natural language processing for requirements engineering: A systematic mapping study, " ACM Computing Surveys, vol. 54, no. 3, pp. 1-41, 2021.
L. Kof, "Scenarios: Identifying missing objects and actions by means of computational linguistics, " in RE'07. IEEE, 2007, pp. 121-130.
B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, "Recent advances in natural language processing via large pre-trained language models: A survey, " ACM Computing Surveys, vol. 56, no. 2, pp. 1-40, 2023.
A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, "Large language models for software engineering: Survey and open problems, " arXiv preprint arXiv: 2310. 03533, 2023.
X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, "Large language models for software engineering: A systematic literature review, " arXiv preprint arXiv: 2308. 10620, 2023.
C. Jain, P. R. Anish, A. Singh, and S. Ghaisas, "A transformer-based approach for abstractive summarization of requirements from obligations in software engineering contracts, " in RE'23. IEEE, 2023, pp. 169-179.
A. D. Rodriguez, K. R. Dearstyne, and J. Cleland-Huang, "Prompts matter: Insights and strategies for prompt engineering in automated software traceability, " in REW'23. IEEE, 2023, pp. 455-464.
K. Ronanki, B. Cabrero-Daniel, J. Horkoff, and C. Berger, "Requirements engineering using generative AI: Prompts and prompting patterns, " arXiv preprint arXiv: 2311. 03832, 2023.
J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, "ChatGPT prompt patterns for improving code quality, refactoring, requirements elicitation, and software design, " arXiv preprint arXiv: 2303. 07839, 2023.
D. Kundu, D. Samanta, and R. Mall, "Automatic code generation from unified modelling language sequence diagrams, " IET Software, vol. 7, no. 1, pp. 12-28, 2013.
A. Ferrari, G. O. Spagnolo, and S. Gnesi, "Pure: A dataset of public requirements documents, " in RE'17. IEEE, 2017, pp. 502-505.
F. Dalpiaz and A. Sturm, "Conceptualizing requirements using user stories and use cases: A controlled experiment, " in REFSQ'20. Springer, 2020, pp. 221-238.
V. Braun and V. Clarke, "To saturate or not to saturate Questioning data saturation as a useful concept for thematic analysis and sample-size rationales, " Qualitative research in sport, exercise and health, vol. 13, no. 2, pp. 201-216, 2021.
J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, "A prompt pattern catalog to enhance prompt engineering with ChatGPT, " arXiv: 2302. 11382, 2023.
D. Zowghi and V. Gervasi, "On the interplay between consistency, completeness, and correctness in requirements evolution, " IST, vol. 45, no. 14, pp. 993-1009, 2003.
"ISO/IEC/IEEE international standard-systems and software engineering-life cycle processes-requirements engineering, " ISO/IEC/IEEE 29148: 2018 (E), pp. 1-104, 2018.
F. Alessio, A. Sallam, and A. Chetan, "Model Generation from Requirements with LLMs: An Exploratory Study-Replication Package, " Apr. 2024. [Online]. Available: https: //doi. org/10. 5281/zenodo. 10579731
J. Cohen, "A coefficient of agreement for nominal scales, " Educational and Psychological Measurement, vol. 20, no. 1, 1960.
S. Abrahão, E. Insfran, J. A. Carsi, and M. Genero, "Evaluating requirements modeling methods based on user perceptions: A family of experiments, " Inf. Sci., vol. 181, no. 16, pp. 3356-3378, 2011.
J. Cohen, Statistical power analysis for the behavioral sciences. Academic press, 2013.
V. Clarke and V. Braun, "Thematic analysis, " The journal of positive psychology, vol. 12, no. 3, pp. 297-298, 2017.
K. Ronanki, C. Berger, and J. Horkoff, "Investigating ChatGPT's potential to assist in requirements elicitation processes, " in SEAA'23. IEEE, 2023, pp. 354-361.
J. Sallou, T. Durieux, and A. Panichella, "Breaking the silence: The threats of using LLMs in software engineering, " arXiv: 2312. 08055, 2023.
