Privacy Requirements; General Data Protection Regulation (GDPR); Natural Language Processing (NLP); Large Language Models (LLMs); Retrieval Augmented Generation (RAG)
Abstract :
[en] Modern software systems increasingly rely on personal data. Despite the enforcement of the European General Data Protection Regulation (GDPR) and the growing awareness about privacy and data protection, many individuals’ rights remain unsatisfactorily implemented in software systems. This is partially due to the knowledge gap between legal interpretation and software development. In this paper, we address this gap first by extracting, in close collaboration with legal experts, a list of 108 requirements pertinent to the right of access (ACC) and the right to portability (PRT), two fundamental rights under the GDPR. We further propose the XTRAREG approach, which utilizes large language models (LLMs) and retrieval augmented generation (RAG) to provide automated assistance in extracting privacy requirements from predefined legal sources. Compared to the manually extracted requirements, XTRAREG can automatically generate requirements with an accuracy of 81.8% for ACC and 56.7% for PRT. Our empirical evaluation reveals two notable observations: (i) A skewed performance in the favor of ACC, indicating the significant impact of abundant training data of the LLM, (ii) despite explicit exposure of legal references through RAG, the LLM generates requirements predominantly from the GDPR.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Disciplines :
Computer science
Author, co-author :
ABUALHAIJA, Sallam ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
CECI, Marcello ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
SANNIER, Nicolas ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
BIANCULLI, Domenico ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
LANNIER, Salomé ; University of Luxembourg > Faculty of Law, Economics and Finance (FDEF) > Department of Law (DL)
SICLARI, Martina ; University of Luxembourg > Faculty of Law, Economics and Finance > Department of Law > Team Stanislaw TOSZA
VOORDECKERS, Olivier ; University of Luxembourg > Faculty of Law, Economics and Finance (FDEF) > Department of Law (DL)
TOSZA, Stanislaw ; University of Luxembourg > Faculty of Law, Economics and Finance (FDEF) > Department of Law (DL)
External co-authors :
no
Language :
English
Title :
LLM-assisted Extraction of Regulatory Requirements: A Case Study on the GDPR
Publication date :
2025
Event name :
33rd IEEE International Requirements Engineering Conference
Event date :
from 1 to 5 September, 2025
Main work title :
Proceedings of the 33rd IEEE International Requirements Engineering Conference (RE'25)
E. Felten, M. Raj, and R. Seamans, "Generative AI requires broad labor policy considerations," Commun. ACM, vol. 67, no. 8, pp. 29-32, 2024.
R. Bhayana, "Chatbots and large language models in radiology: a practical primer for clinical and research applications," Radiology, vol. 310, no. 1, p. e232756, 2024.
W. Rong and Z. Yu, "Do AI chatbots improve students learning outcomes? evidence from a meta-analysis," Br. J. Educ. Technol., vol. 55, no. 1, pp. 10-33, 2024.
W. Seymour, X. Zhan, M. Coté, and J. M. Such, "A systematic review of ethical concerns with voice assistants," in Proceedings of AIES 2023. ACM, 2023, pp. 131-145.
European Union, Charter of Fundamental Rights of the European Union. Brussels: European Union, 2010, vol. 53.
The European Parliament and the Council of the European Union, "Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation)," 05 2016.
T. D. Breaux, M. W. Vail, and A. I. Antón, "Towards regulatory compliance: Extracting rights and obligations to align requirements with regulations," in Proceedings of RE 2006. IEEE, 2006, pp. 46-55.
S. Ghanavati, D. Amyot, and A. Rifaut, "Legal goal-oriented requirement language (legal GRL) for modeling regulations," in Proceedings of MiSE 2014. ACM, 2014, pp. 1-6.
N. Zeni, N. Kiyavitskaya, L. Mich, J. R. Cordy, and J. Mylopoulos, "Gaiust: supporting the extraction of rights and obligations for regulatory compliance," Requir. Eng., vol. 20, no. 1, pp. 1-22, 2015.
A. Sleimi, N. Sannier, M. Sabetzadeh, L. C. Briand, M. Ceci, and J. Dann, "An automated framework for the extraction of semantic legal metadata from legal texts," Empir. Softw. Eng., vol. 26, no. 3, p. 43, 2021.
J. Tom, E. Sing, and R. Matulevicius, "Conceptual representation of the GDPR: model and application directions," in Proceedings of BIR 2018, ser. Lecture Notes in Business Information Processing, vol. 330. Springer, 2018, pp. 18-28.
D. Torre, M. Alférez, G. Soltana, M. Sabetzadeh, and L. C. Briand, "Modeling data protection and privacy: application and experience with GDPR," Softw. Syst. Model., vol. 20, no. 6, pp. 2071-2087, 2021.
O. Amaral, S. Abualhaija, D. Torre, M. Sabetzadeh, and L. C. Briand, "AI-enabled automation for completeness checking of privacy policies," IEEE Trans. Software Eng., vol. 48, no. 11, pp. 4647-4674, 2022.
M. I. Azeem and S. Abualhaija, "A multi-solution study on GDPR AIenabled completeness checking of dpas," Empir. Softw. Eng., vol. 29, no. 4, p. 96, 2024.
M. Fan, L. Yu, S. Chen, H. Zhou, X. Luo, S. Li, Y. Liu, J. Liu, and T. Liu, "An empirical evaluation of GDPR compliance violations in android mhealth apps," in Proceedings of ISSRE 2020. IEEE, 2020, pp. 253-264.
M. Hatamian, S. Wairimu, N. Momen, and L. Fritsch, "A privacy and security analysis of early-deployed COVID-19 contact tracing android apps," Empir. Softw. Eng., vol. 26, no. 3, p. 36, 2021.
C. Negri-Ribalta and M. L.-P. C. Salinesi, "Understanding the GDPR from a requirements engineering perspective-a systematic mapping study on regulatory data protection requirements," Requir. Eng., pp. 1-27, 2024.
noyb-European Center for Digital Rights, "GDPR: a culture of non-compliance? Numbers of evidence-based enforcement efforts," Accessed Sep. 24, 2024 [Online], 2024. [Online]. Available: https://noyb.eu/sites/default/files/2024-01/GDPR a%20culture%20of%20non-compliance 2.pdf
Article 29 Data Protection Working Party, "Guidelines on the right to data portability," 2017. [Online]. Available: https://ec.europa.eu/ newsroom/article29/items/611233
B. Görer and F. B. Aydemir, "Generating requirements elicitation interview scripts with large language models," in Proceedings of RE 2023-Workshops. IEEE, 2023, pp. 44-51.
D. Bernsohn, G. Semo, Y. Vazana, G. Hayat, B. Hagag, J. Niklaus, R. Saha, and K. Truskovskyi, "Legallens: Leveraging llms for legal violation identification in unstructured text," in Proceedings of EACL 2024. ACL, 2024, pp. 2129-2145.
F. Contini, "Unboxing generative AI for the legal professions: Functions, impacts and governance," International Journal for Court Administration, vol. 15, no. 2, 2024.
A. S. Kwak, C. Jeong, G. Forte, D. E. Bambauer, C. T. Morrison, and M. Surdeanu, "Information extraction from legal wills: How well does GPT-4 do?" in Proceedings of EMNLP 2023. ACL, 2023, pp. 4336-4353.
F. Yu, L. Quartey, and F. Schilder, "Exploring the effectiveness of prompt engineering for legal reasoning tasks," in Proceedings of ACL 2023. ACL, 2023, pp. 13 582-13 596.
P. S. H. Lewis and others, "Retrieval-augmented generation for knowledge-intensive NLP tasks," in Proceedings of NeurIPS 2020, 2020.
S. Abualhaija, M. Ceci, N. Sannier, D. Bianculli, S. Lannier, M. Siclari, O. Voordeckers, and S. Tosza, "Online Annex to the paper: "LLMassisted Elicitation of Regulatory Requirements: A Case Study on the GDPR"," https://doi.org/10.6084/m9.figshare.27187557, 2025.
-, "Artifact associated with "LLM-assisted extraction of regulatory requirements: A case study on the GDPR"," https://doi.org/10.5281/zenodo.15668459, 2025.
R. Gellert and S. Gutwirth, "The legal construction of privacy and data protection," Comput. Law Secur. Rev., vol. 29, no. 5, pp. 522-530, 2013.
The European Parliament and the Council of the European Union, "Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data," 10 1995.
P. A. Chitale, J. P. Gala, and R. Dabre, "An empirical study of in-context learning in llms for machine translation," in Proceedingns of ACL 2024. ACL, 2024, pp. 7384-7406.
D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed. Prentice Hall, 2020.
C. Arora, J. Grundy, and M. Abdelrazek, Advancing Requirements Engineering Through Generative AI: Assessing the Role of LLMs. Springer Nature Switzerland, 2024, pp. 129-148.
S. Abualhaija, M. Ceci, N. Sannier, D. Bianculli, L. C. Briand, D. A. Zetzsche, and M. Bodellini, "AI-enabled regulatory change analysis of legal requirements," in Proceedings of RE 2024. IEEE, 2024, pp. 5-17.
K. Ronanki, B. Cabrero-Daniel, J. Horkoff, and C. Berger, Requirements Engineering Using Generative AI: Prompts and Prompting Patterns. Springer, 2024, pp. 109-127.
S. Lubos, A. Felfernig, T. N. T. Tran, D. Garber, M. E. Mansi, S. P. Erdeniz, and V. Le, "Leveraging llms for the quality assurance of software requirements," in Proceedings of RE 2024. IEEE, 2024, pp. 389-397.
T. B. Brown and others, "Language models are few-shot learners," in Proceedings of NeurIPS 2020, 2020.
T. Schick and H. Schütze, "Exploiting cloze-questions for few-shot text classification and natural language inference," in Proceedings of EACL 2021. ACL, 2021, pp. 255-269.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, "Chain-of-thought prompting elicits reasoning in large language models," in Proceedings of NeurIPS 2022, 2022.
The European Parliament and the Council of the European Union, "Regulation (EU) 2023/2854 of the European Parliament and of the Council of 13 December 2023 on harmonised rules on fair access to and use of data and amending Regulation (EU) 2017/2394 and Directive (EU) 2020/1828 (Data Act)," 12 2023.
P. de Hert, V. Papakonstantinou, G. Malgieri, L. Beslay, and I. Sánchez, "The right to data portability in the GDPR: towards user-centric interoperability of digital services," Comput. Law Secur. Rev., vol. 34, no. 2, pp. 193-203, 2018.
Article 29 Data Protection Working Party, "Opinion 02/2013 on apps on smart devices," 2013. [Online]. Available: https://ec.europa.eu/justice/article-29/documentation/ opinion-recommendation/files/2013/wp202 en.pdf
European Data Protection Board, "Guidelines 01/2022 on data subject right-right of access," 2022. [Online]. Available: https://www.edpb.europa.eu/our-work-tools/our-documents/ guidelines/guidelines-012022-data-subject-rights-right-access en
European Court of Justice, "Joined cases c-141/12 and c-372/12: Judgment of the court (third chamber) of 17 july 2014; ecli:eu:c:2023:369; celex:62012ca0141," 2014.
-, "Judgment of the court (first chamber) of 4 may 2023; case c-487/21; ecli:eu:c:2023:369; celex:62021cj0487," 2023.
F. Sovrano, M. Lognoul, and A. Bacchelli, "An empirical study on compliance with ranking transparency in the software documentation of EU online platforms," in Proceedings of ICSE-SEIS'2024. ACM, 2024, pp. 46-56.
S. Shin and Y. Kim, "Enhancing graph of thought: Enhancing prompts with LLM rationales and dynamic temperature control," in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=l32IrJtpOP
S. Sanyal, T. Xiao, J. Liu, W. Wang, and X. Ren, "Are machines better at complex reasoning? unveiling human-machine inference gaps in entailment verification," in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 10 361-10 386.
E. M. Smith, O. Hsu, R. Qian, S. Roller, Y. Boureau, and J. Weston, "Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents," in Proceedings of ConvAI@ACL 2022. ACL, 2022, pp. 77-97.
C. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau, "How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation," in Proceedings of EMNLP 2016,. ACL, 2016, pp. 2122-2132.
P. Manakul, A. Liusie, and M. J. F. Gales, "Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models," in Proceedings of EMNLP 2023. ACL, 2023, pp. 9004-9017.
L. Zheng and others, "Judging llm-as-a-judge with mt-bench and chatbot arena," in Proceedings of NeurIPS' 2023, 2023.
X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, "Deep code comment generation with hybrid lexical and syntactical information," Empirical Software Engineering, vol. 25, pp. 2179-2217, 2020.
L. Kuang, C. Zhou, and X. Yang, "Code comment generation based on graph neural network enhanced transformer model for code understanding in open-source software ecosystems," Autom. Softw. Eng., vol. 29, no. 2, p. 43, 2022.
S. Ezzini, S. Abualhaija, C. Arora, and M. Sabetzadeh, "AI-based question answering assistance for analyzing natural-language requirements," in Proceedings of ICSE 2023. IEEE, 2023, pp. 1277-1289.
S. Xu, L. Pang, M. Yu, F. Meng, H. Shen, X. Cheng, and J. Zhou, "Unsupervised information refinement training of large language models for retrieval-augmented generation," in Proceedings of ACL 2024. ACL, 2024, pp. 133-145.
K. Papineni, S. Roukos, T. Ward, and W. Zhu, "Bleu: a method for automatic evaluation of machine translation," in Proceedings of ACL 2002. ACL, 2002, pp. 311-318.
C.-Y. Lin, "ROUGE: A package for automatic evaluation of summaries," in Text Summarization Branches Out. ACL, 07 2004, pp. 74-81.
S. Banerjee and A. Lavie, "METEOR: an automatic metric for MT evaluation with improved correlation with human judgments," in Proceedings of IEEvaluation 2005. ACL, 2005, pp. 65-72.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "Bertscore: Evaluating text generation with BERT," in Proceedings of ICLR 2020. OpenReview.net, 2020.
V. Magesh, F. Surani, M. Dahl, M. Suzgun, C. D. Manning, and D. E. Ho, "Hallucination-free? assessing the reliability of leading AI legal research tools," CoRR, vol. abs/2405.20362, 2024.
M. Dahl, V. Magesh, M. Suzgun, and D. E. Ho, "Large legal fictions: Profiling legal hallucinations in large language models," Journal of Legal Analysis, vol. 16, no. 1, pp. 64-93, 2024.
L. Humphreys, C. Santos, L. Di Caro, G. Boella, L. Van Der Torre, and L. Robaldo, "Mapping recitals to normative provisions in eu legislation to assist legal interpretation," in Legal Knowledge and Information Systems. IOS Press, 2015, pp. 41-49.
L. Chen, M. Zaharia, and J. Zou, "How Is ChatGPT's Behavior Changing Over Time?" Harvard Data Science Review, vol. 6, no. 2, 2024.
J. Bhatia, M. C. Evans, and T. D. Breaux, "Identifying incompleteness in privacy policy goals using semantic frames," Requir. Eng., vol. 24, no. 3, pp. 291-313, 2019.
O. Dieste and N. J. Juzgado, "Systematic review and aggregation of empirical studies on elicitation techniques," IEEE Trans. Software Eng., vol. 37, no. 2, pp. 283-304, 2011.
A. G. Sutcliffe and P. Sawyer, "Requirements elicitation: Towards the unknown unknowns," in Proceedings of RE 2013. IEEE, 2013, pp. 92-104.
T. D. Breaux and A. I. Antón, "Analyzing regulatory rules for privacy and security requirements," IEEE Trans. Software Eng., vol. 34, no. 1, pp. 5-20, 2008.
A. K. Massey, P. N. Otto, and A. I. Antón, "Aligning requirements with HIPAA in the itrust system," in Proceedings of RE 2008. IEEE, 2008, pp. 335-336.
K. Kolthoff, C. Bartelt, S. P. Ponzetto, and K. Schneider, "Self-elicitation of requirements with automated gui prototyping," in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 2354-2357.
J. Wei, A.-L. Courbis, T. Lambolais, B. Xu, P. L. Bernard, G. Dray, and W. Maalej, "Getting inspiration for feature elicitation: App storevs. llm-based approach," in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '24. Association for Computing Machinery, 2024, pp. 857-869.
S. Ren, H. Nakagawa, and T. Tsuchiya, "Combining prompts with examples to enhance llm-based requirement elicitation," in 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 2024, pp. 1376-1381.
M. R. Tabassum, M. J. Ritchie, S. Mustafiz, and J. Kienzle, "Using llms for use case modelling of iot systems: An experience report," in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems, 2024, pp. 611-619.
G. A. Morales, P. K. C, S. Jahan, M. B. Hosseini, and R. Slavin, "A large language model approach to code and privacy policy alignment," in Proceedings of SANER 2024. IEEE, 2024, pp. 79-90.
D. Rodriguez, I. Yang, J. M. Del Alamo, and N. Sadeh, "Large language models: a new approach for privacy policy analysis at scale," Computing, vol. 106, no. 12, pp. 3879-3903, 2024.
J. Ioannidis, J. Harper, M. S. Quah, and D. Hunter, "Gracenote. ai: Legal generative ai for regulatory compliance," in Proceedings of the Third International Workshop on Artificial Intelligence and Intelligent Assistance for Legal Professionals in the Digital Workplace (LegalAIIA 2023), 2023.
K. Ronanki, C. Berger, and J. Horkoff, "Investigating chatgpt's potential to assist in requirements elicitation processes," in Proceedings of SEAA 2023. IEEE, 2023, pp. 354-361.
