Regulatory Change; Prompt Engineering; Natural Language Processing (NLP); Large Language Models (LLMs); ChatGPT; Regulatory Compliance
Abstract :
[en] Statutory law is subject to change as legislation
develops over time – new regulation can be introduced, while
existing regulation can be amended, or repealed. From a requirements
engineering (RE) perspective, such change must be dealt
with to ensure the compliance of software systems at all times.
Understanding the implications of regulatory change on compliance
of software requirements requires navigating hundreds
of legal provisions. Analyzing instances of regulatory change
entirely manually is not only time-consuming, but also risky, since
missing a change may result in non-compliant software which can
in turn lead to hefty fines. In this paper, we propose MURCIA,
an automated approach that leverages recent language models to
assist human analysts in analyzing regulatory changes. To build
MURCIA, we define a taxonomy that characterizes the regulatory
changes at the textual level as well as the changes in the text’s
meaning and legal interpretation. We evaluate MURCIA on four
regulations from the financial domain. Over our evaluation set,
MURCIA can identify textual changes with F1 score of 90.5%,
and it can provide, according to our taxonomy, the text meaning
and legal interpretation with an F1 score of 90.8% and 83.7%, respectively.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation NCER-FT - FinTech National Centre of Excellence in Research
Disciplines :
Computer science
Author, co-author :
ABUALHAIJA, Sallam ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
CECI, Marcello ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
SANNIER, Nicolas ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
BIANCULLI, Domenico ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Briand, Lionel; Lero SFI centre for Software Research and University of Limerick ; University of Ottawa > School of EECS
ZETZSCHE, Dirk Andreas ; University of Luxembourg > Faculty of Law, Economics and Finance (FDEF) > Department of Law (DL)
BODELLINI, Marco ; University of Luxembourg > Faculty of Law, Economics and Finance (FDEF) > Department of Law (DL)
External co-authors :
yes
Language :
English
Title :
AI-enabled Regulatory Change Analysis of Legal Requirements
Publication date :
2024
Event name :
32nd IEEE International Requirements Engineering Conference
Event place :
Reykjavik, Iceland
Event date :
from 24 to 28 June, 2024
Audience :
International
Main work title :
Proceedings of the 32nd IEEE International Requirements Engineering Conference (RE'24)
Publisher :
IEEE
Pages :
5-17
Peer reviewed :
Peer reviewed
FnR Project :
FNR16570468 - 2021 (01/07/2022-30/06/2030) - Yves Le Traon
S. Abualhaija, C. Arora, A. Sleimi, and L. C. Briand, "Automated question answering for improved understanding of compliance requirements: A multi-document study," in 30th IEEE International Requirements Engineering Conference, RE 2022. IEEE, 2022, pp. 39-50.
M. Robol, T. D. Breaux, E. Paja, and P. Giorgini, "Consent verification monitoring," ACM Trans. Softw. Eng. Methodol., vol. 32, no. 1, 2023.
E. Johansson, K. Sutinen, J. Lassila, V. Lang, M. Martikainen, and O. M. Lehner, "Regtech-a necessary tool to keep up with compliance and regulatory changes," ACRN Journal of Finance and Risk Perspectives, Special Issue Digital Accounting, vol. 8, pp. 71-85, 2019.
S. Abualhaija, M. Ceci, and L. Briand, "Legal requirements analysis," arXiv preprint arXiv:2311.13871, 2023.
D. W. Arner, J. Barberis, and R. P. Buckley, "The evolution of fintech: A new post-crisis paradigm," Geo. J. Int'l L., vol. 47, p. 1271, 2015.
Committee of European Securities Regulators, "CESR's Guidelines on Risk Measurement and the Calculation of Global Exposure and Counterparty Risk for UCITS," 07 2010, https://www.esma.europa.eu/ sites/default/files/library/2015/11/10 788.pdf.
European Securities and Markets Authority (ESMA), "ESMA's Technical Advice to the Commission on the effects of product intervention measures," 02 2020, https://www.esma.europa.eu/sites/ default/files/library/esma35-43-2134 technical advice to the ec on product intervention.pdf.
USA Government-Executive Office of the President, "88 FR 54867-Addressing United States Investments in Certain National Security Technologies and Products in Countries of Concern," 08 2023, https: //www.federalregister.gov/d/2023-17449.
The European Parliament and Council, "Directive 2011/61/EU of the European Parliament and of the Council of 8 June 2011 on Alternative Investment Fund Managers and amending Directives 2003/41/EC and 2009/65/EC and Regulations (EC) No 1060/2009 and (EU) No 1095/2010," 2011. [Online]. Available: https: //eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32011L0061.
S. Abualhaija, M. Ceci, N. Sannier, D. Bianculli, D. Zetzsche, and M. Bodellini, "Toward automated change impact analysis of financial regulations," in 2024 IEEE/ACM 1st 1st Workshop on Software Engineering Challenges in Financial Firms (FinanSE). IEEE, 2024.
A. Sleimi, N. Sannier, M. Sabetzadeh, L. C. Briand, M. Ceci, and J. Dann, "An automated framework for the extraction of semantic legal metadata from legal texts," Empir. Softw. Eng., vol. 26, no. 3, p. 43, 2021.
T. D. Breaux and A. I. Antón, "Analyzing regulatory rules for privacy and security requirements," IEEE Trans. Software Eng., vol. 34, no. 1, pp. 5-20, 2008.
N. Zeni, N. Kiyavitskaya, L. Mich, J. R. Cordy, and J. Mylopoulos, "Gaiust: supporting the extraction of rights and obligations for regulatory compliance," Requir. Eng., vol. 20, no. 1, pp. 1-22, 2015.
S. Ben Nasr, N. Sannier, M. Acher, and B. Baudry, "Moving toward product line engineering in a nuclear industry consortium," in 18th International Software Product Line Conference, SPLC '14. ACM, 2014, pp. 294-303.
D. G. Gordon and T. D. Breaux, "Reconciling multi-jurisdictional legal requirements: A case study in requirements water marking," in 2012 20th IEEE International Requirements Engineering Conference (RE). IEEE Computer Society, 2012, pp. 91-100.
G. Soltana, N. Sannier, M. Sabetzadeh, and L. C. Briand, "Model-based simulation of legal policies: framework, tool support, and validation," Softw. Syst. Model., vol. 17, no. 3, pp. 851-883, 2018.
A. Rabinia, S. Ghanavati, L. Humphreys, and T. Hahmann, "A methodology for implementing the formal legal-grl framework: A research preview," in Requirements Engineering: Foundation for Software Quality-26th International Working Conference, REFSQ 2020, Pisa, Italy, March 24-27, 2020, Proceedings, ser. Lecture Notes in Computer Science, vol. 12045. Springer, 2020, pp. 124-131.
D. Torre, M. Alférez, G. Soltana, M. Sabetzadeh, and L. C. Briand, "Modeling data protection and privacy: application and experience with GDPR," Softw. Syst. Model., vol. 20, no. 6, pp. 2071-2087, 2021.
O. Amaral, S. Abualhaija, M. Sabetzadeh, and L. C. Briand, "A modelbased conceptualization of requirements for compliance checking of data processing against GDPR," in 29th IEEE International Requirements Engineering Conference Workshops, RE 2021 Workshops, 2021, pp. 16-20.
T. D. Breaux and D. G. Gordon, "Regulatory requirements traceability and analysis using semi-formal specifications," in Proceedings of Requirements Engineering: Foundation for Software Quality-19th International Working Conference, REFSQ 2013. Springer, 2013, pp. 141-157.
A. Parvizimosaed, S. Sharifi, D. Amyot, L. Logrippo, M. Roveri, A. Rasti, A. Roudak, and J. Mylopoulos, "Specification and analysis of legal contracts with symboleo," Softw. Syst. Model., vol. 21, no. 6, pp. 2395-2427, 2022.
D. G. Gordon and T. D. Breaux, "Assessing regulatory change through legal requirements coverage modeling," in 21st IEEE International Requirements Engineering Conference, RE 2013. IEEE Computer Society, 2013, pp. 145-154.
R. Joseph, T. Liu, A. B. Ng, S. See, and S. Rai, "Newsmet: A 'do it all' dataset of contemporary metaphors in news headlines," in Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023, pp. 10 090-10 104.
V. Iyer, P. Chen, and A. Birch, "Towards effective disambiguation for machine translation with large language models," in Proceedings of the Eighth Conference on Machine Translation, WMT 2023. Association for Computational Linguistics, 2023, pp. 482-495.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
S. Abualhaija, M. Ceci, N. Sannier, D. Bianculli, L. Briand, D. Zetzsche, and M. Bodellini, "Online Annex (online)", 2024, DOI: 10.5281/zenodo. 10959496, available at https://doi.org/10.5281/zenodo.10959496.
D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed. Prentice Hall, 2020.
A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, "Superglue: A stickier benchmark for generalpurpose language understanding systems," Advances in neural information processing systems, vol. 32, 2019.
T. Schick and H. Schütze, "Exploiting cloze-questions for few-shot text classification and natural language inference," in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021, pp. 255-269.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in Neural Information Processing Systems, vol. 35, pp. 24 824-24 837, 2022.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transformers for language understanding," CoRR, vol. abs/1810.04805, 2018.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models are zero-shot reasoners," Advances in neural information processing systems, vol. 35, pp. 22 199-22 213, 2022.
P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, "Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing," ACM Computing Surveys, vol. 55, no. 9, pp. 1-35, 2023.
F. Yu, L. Quartey, and F. Schilder, "Exploring the effectiveness of prompt engineering for legal reasoning tasks," in Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 13 582-13 596.
M. Grüninger and M. S. Fox, "The role of competency questions in enterprise engineering," in Benchmarking-Theory and practice. Springer, 1995, pp. 22-31.
The European Commission, "Commission Delegated Regulation (EU) No 231/2013 of 19 December 2012 supplementing Directive 2011/61/EU of the European Parliament and of the Council with regard to exemptions, general operating conditions, depositaries, leverage, transparency and supervision," 2013. [Online]. Available: https://eur-lex. europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32011L0061.
G. H. Von Wright, On the Logic of Norms and Actions. Springer Netherlands, 1981, pp. 3-35.
G. Boella and L. van der Torre, "Regulative and constitutive norms in normative multiagent systems," in Proceedings of the Ninth International Conference on Principles of Knowledge Representation and Reasoning, ser. KR'04. AAAI Press, 2004, p. 255-265.
M. Ceci, T. Butler, L. O'Brien, and F. A. Khalil, "Legal patterns for different constitutive rules," in AI Approaches to the Complexity of Legal Systems-AICOL International Workshops 2015-2017: AICOLVI@ JURIX 2015, AICOL-VII@EKAW 2016, AICOL-VIII@JURIX 2016, AICOL-IX@ICAIL 2017, and AICOL-X@JURIX 2017, Revised Selected Papers, ser. Lecture Notes in Computer Science, vol. 10791. Springer, 2017, pp. 105-123.
R. Hoekstra, J. Breuker, M. D. Bello, and A. Boer, "The LKIF core ontology of basic legal concepts," in Proceedings of the 2nd Workshop on Legal Ontologies and Artificial Intelligence Techniques June 4th, 2007, Stanford University, Stanford, CA, USA, ser. CEUR Workshop Proceedings, vol. 321. CEUR-WS.org, 2007, pp. 43-63.
F. Al Khalil, M. Ceci, K. Yapa, and L. O'Brien, "SBVR to OWL 2 mapping in the domain of legal rules," in Rule Technologies. Research, Tools, and Applications: 10th International Symposium, RuleML 2016, Stony Brook, NY, USA, July 6-9, 2016. Proceedings 10. Springer, 2016, pp. 258-266.
A. Ciabattoni, X. Parent, and G. Sartor, "Permission in a kelsenian perspective," in Legal Knowledge and Information Systems-JURIX 2022: The Thirty-sixth Annual Conference, Maastricht, the Netherlands, 18-20 December 2023. IOS Press, 12 2023.
T. Kluyver, B. Ragan-Kelley, F. Pérez, B. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. Hamrick, J. Grout, S. Corlay, P. Ivanov, D. Avila, S. Abdalla, and C. Willing, "Jupyter notebooks-a publishing format for reproducible computational workflows," in Positioning and Power in Academic Publishing: Players, Agents and Agendas, 2016.
E. Loper and S. Bird, "NLTK: The natural language toolkit," in Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. ACL, 2002, pp. 62-69.
"The difflib module," 2023. [Online]. Available: https://docs.python. org/3/library/difflib.html.
J. Cohen, "A coefficient of agreement for nominal scales," Educational and Psychological Measurement, vol. 20, no. 1, 1960.
D. M. Berry, "Requirements engineering for artificial intelligence: What is a requirements specification for an artificial intelligence?" in Requirements Engineering: Foundation for Software Quality-28th International Working Conference, REFSQ 2022 Proceedings, ser. Lecture Notes in Computer Science, vol. 13216. Springer, 2022, pp. 19-25.
S. Harker, K. D. Eason, and J. E. Dobson, "The change and evolution of requirements as a challenge to the practice of software engineering," in Proceedings of IEEE International Symposium on Requirements Engineering, RE 1993. IEEE Computer Society, 1993, pp. 266-272.
D. Zowghi and R. Offen, "A logical framework for modeling and reasoning about the evolution of requirements," in 3rd IEEE International Symposium on Requirements Engineering (RE'97). IEEE Computer Society, 1997, p. 247.
C. Arora, M. Sabetzadeh, A. Goknil, L. C. Briand, and F. Zimmer, "Change impact analysis for natural language requirements: An nlp approach," in 2015 IEEE 23rd International Requirements Engineering Conference (RE). IEEE, 2015, pp. 6-15.
S. Nejati, M. Sabetzadeh, C. Arora, L. C. Briand, and F. Mandoux, "Automated change impact analysis between sysml models of requirements and design," in Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016. ACM, 2016, pp. 242-253.
N. Zeni, E. A. Seid, P. Engiel, and J. Mylopoulos, "NómosT: Building large models of law with a tool-supported process," Data Knowl. Eng., vol. 117, pp. 407-418, 2018.
T. D. Breaux and T. B. Norton, "Legal accountability as software quality: A U.S. data processing perspective," in 30th IEEE International Requirements Engineering Conference, RE 2022. IEEE, 2022, pp. 101-113.
J. C. Maxwell, A. I. Antón, and P. P. Swire, "Managing changing compliance requirements by predicting regulatory evolution," in 2012 20th IEEE International Requirements Engineering Conference (RE), Chicago, IL, USA, September 24-28, 2012. IEEE Computer Society, 2012, pp. 101-110.
S. Saito, Y. Iimura, K. Takahashi, A. K. Massey, and A. I. Antón, "Tracking requirements evolution by using issue tickets: a case study of a document management and approval system," in 36th International Conference on Software Engineering, ICSE '14, Companion Proceedings. ACM, 2014, pp. 245-254.
S. Saito, Y. Iimura, H. Tashiro, A. K. Massey, and A. I. Antón, "Visualizing the effects of requirements evolution," in Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, Companion Volume. ACM, 2016, pp. 152-161.
D. G. Gordon and T. D. Breaux, "Managing multi-jurisdictional requirements in the cloud: towards a computational legal landscape," in Proceedings of the 3rd ACM Cloud Computing Security Workshop, CCSW 2011, Chicago, IL, USA, October 21, 2011. ACM, 2011, pp. 83-94.
I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos, "LEGAL-BERT: The muppets straight out of law school," in Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, 2020, pp. 2898-2904.
H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun, "How does NLP benefit legal system: A summary of legal artificial intelligence," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020, pp. 5218-5230.
N. Holzenberger and B. Van Durme, "Connecting symbolic statutory reasoning with legal information extraction," in Proceedings of the Natural Legal Language Processing Workshop 2023. Association for Computational Linguistics, 2023, pp. 113-131.
J. Soh, H. K. Lim, and I. E. Chai, "Legal area classification: A comparative study of text classifiers on Singapore Supreme Court judgments," in Proceedings of the Natural Legal Language Processing Workshop 2019. Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 67-77.
E. Leitner, G. Rehm, and J. Moreno-Schneider, "Fine-grained named entity recognition in legal documents," in International Conference on Semantic Systems. Springer, 2019, pp. 272-287.
M. Mistica, G. Z. Zhang, H. Chia, K. M. Shrestha, R. K. Gupta, S. Khandelwal, J. Paterson, T. Baldwin, and D. Beck, "Information extraction from legal documents: A study in the context of common law court judgements," in Proceedings of the 18th Annual Workshop of the Australasian Language Technology Association. Virtual Workshop: Australasian Language Technology Association, 2020, pp. 98-103. [Online]. Available: https://aclanthology.org/2020.alta-1.12.
R. Shui, Y. Cao, X. Wang, and T.-S. Chua, "A comprehensive evaluation of large language models on legal judgment prediction," in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 7337-7348.
A. Kwak, C. Jeong, G. Forte, D. Bambauer, C. Morrison, and M. Surdeanu, "Information extraction from legal wills: How well does GPT-4 do?" in Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, Dec. 2023, pp. 4336-4353.
J. Daxenberger and I. Gurevych, "A corpus-based study of edit categories in featured and non-featured wikipedia articles," in COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers. Indian Institute of Technology Bombay, 2012, pp. 711-726.
-, "Automatically classifying edit categories in wikipedia revisions," in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, 2013, pp. 578-589.
D. Yang, A. Halfaker, R. Kraut, and E. Hovy, "Identifying semantic edit intentions from revisions in wikipedia," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 2000-2010.
A. Spangher, X. Ren, J. May, and N. Peng, "Newsedits: A news article revision dataset and a novel document-level reasoning challenge," in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022. Association for Computational Linguistics, 2022, pp. 127-157.
W. Du, V. Raheja, D. Kumar, Z. M. Kim, M. Lopez, and D. Kang, "Understanding iterative revision from human-written text," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 3573-3590.
