Requirements Engineering; Regulatory Compliance; Natural Language Processing (NLP); Question Answering; Language Models (LMs); BERT
Abstract :
[en] Software systems are increasingly subject to regulatory compliance. Extracting compliance requirements from regulations is challenging. Ideally, locating compliance-related information in a regulation requires a joint effort from requirements engineers and legal experts, whose availability is limited. However, regulations are typically long documents spanning hundreds of pages, containing legal jargon, applying complicated natural language structures, and including cross-references,
thus making their analysis effort-intensive. In this paper, we propose an automated question-answering (QA) approach that assists requirements engineers in finding the legal text passages relevant to compliance requirements. Our approach utilizes large-scale language models fine-tuned for QA, including BERT and three variants. We evaluate our approach on 107 question-answer pairs, manually curated by subject-matter experts, for four different European regulatory documents. Among these documents is the general data protection regulation (GDPR) – a major source for privacy-related requirements. Our empirical results show that, in ~94% of the cases, our approach finds the text passage containing the answer to a given question among the top five passages that our approach marks as most relevant. Further, our approach successfully demarcates, in the selected passage, the right answer with an average accuracy of ~ 91%.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Disciplines :
Computer science
Author, co-author :
Abualhaija, Sallam ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Arora, Chetan; Deakin University
Sleimi, Amin; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Briand, Lionel ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
External co-authors :
yes
Language :
English
Title :
Automated Question Answering for Improved Understanding of Compliance Requirements: A Multi-Document Study
Publication date :
2022
Event name :
30th IEEE International Requirements Engineering Conference
Event date :
from 15-0802022 to 19-08-2022
Main work title :
Proceedings of the 30th IEEE International Requirements Engineering Conference (RE'22), Melbourne, Australia 15-19 August 2022
Publisher :
IEEE
Pages :
39-50
Peer reviewed :
Peer reviewed
FnR Project :
FNR11801776 - Semantic Metadata And Compliance Rule Extraction From Legal Texts, 2017 (01/01/2018-30/04/2021) - Lionel Briand
P. Klaus and R. Chris, Requirements Engineering Fundamentals, 1st ed. Rocky Nook, 2011.
D. E. Leidner and O. Tona, "The care theory of dignity amid personal data digitalization. " MIS Quarterly, vol. 45, no. 1, 2021.
EU (2019/770), "Directive (EU) 2019/770 of the European Parliament and of the Council of 20 May 2019 on certain aspects concerning contracts for the supply of digital content and digital services, OJ L 136, 22. 5. 2019, p. 1-27, " 2019. [Online]. Available: Http://data. europa. eu/eli/dir/2019/770/oj
A. Sleimi, M. Ceci, N. Sannier, M. Sabetzadeh, L. Briand, and J. Dann, "A query system for extracting requirementsrelated information from legal texts, " in 27th IEEE International Requirements Engineering Conference. IEEE, 2019.
A. Sleimi, N. Sannier, M. Sabetzadeh, L. Briand, and J. Dann, "Automated extraction of semantic legal metadata using natural language processing, " in Proceedings of the 26th IEEE International Requirements Engineering Conference, 2018.
A. Ravichander, A. W. Black, S. Wilson, T. Norton, and N. Sadeh, "Question answering for privacy policies: Combining computational and legal perspectives, " in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019.
P. Delfino, B. Cuconato, G. Paulino-Passos, G. Zaverucha, and A. Rademaker, "Using OpenWordnet-PT for question answering on legal domain, " in Proceedings of the 9th Global Wordnet Conference. Global Wordnet Association, 2018.
P. M. Kien, H.-T. Nguyen, N. X. Bach, V. Tran, M. Le Nguyen, and T. M. Phuong, "Answering legal questions by learning neural attentive text representation, " in Proceedings of the 28th International Conference on Computational Linguistics, 2020.
S. Khazaeli, J. Punuru, C. Morris, S. Sharma, B. Staub, M. Cole, S. Chiu-Webster, and D. Sakalley, "A free format legal question answering system, " in Proceedings of the Natural Legal Language Processing Workshop, 2021, pp. 107-113.
D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed., 2020, https://web. stanford. edu/?jurafsky/slp3/ (visited on 2022-01-04).
Z. Zhang, H. Zhao, and R. Wang, "Machine reading comprehension: The role of contextualized language models and beyond, " arXiv preprint arXiv:2005. 06249, 2020.
S. Abualhaija, C. Arora, A. Sleimi, and L. Briand, "Online Annex (online)", 2022, available at shorturl. At/ETX17, February 2022.
A. Alexandrescu and K. Kirchhoff, "Factored neural language models, " in Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, 2006, pp. 1-4.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding, " CoRR, vol. Abs/1810. 04805, 2018.
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, "Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, " in IEEE International Conference on Computer Vision, 2015, pp. 19-27.
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "ALBERT: A lite BERT for self-supervised learning of language representations, " CoRR, vol. Abs/1909. 11942, 2019.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "RoBERTa: A robustly optimized BERT pretraining approach, " CoRR, vol. Abs/1907. 11692, 2019.
S. Nagel, "Cc-news, " 2016. [Online]. Available: Https: //commoncrawl. org/2016/10/news-dataset-available/
A. Gokaslan and V. Cohen, "Openwebtext corpus, " 2019. [Online]. Available: Http://Skylion007. github. io/
T. H. Trinh and Q. V. Le, "A simple method for commonsense reasoning, " CoRR, vol. Abs/1806. 02847, 2018. [Online]. Available: Http://arxiv. org/abs/1806. 02847
K. Clark, M. Luong, Q. V. Le, and C. D. Manning, "ELECTRA: Pre-training text encoders as discriminators rather than generators, " CoRR, vol. Abs/2003. 10555, 2020.
N. Reimers and I. Gurevych, "Sentence-bert: Sentence embeddings using siamese bert-networks, " CoRR, vol. Abs/1908. 10084, 2019.
C. Zeng, S. Li, Q. Li, J. Hu, and J. Hu, "A survey on machine reading comprehension-tasks, evaluation metrics and benchmark datasets, " Applied Sciences, vol. 10, no. 21, p. 7640, 2020.
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "Squad: 100, 000+ questions for machine comprehension of text, " CoRR, vol. Abs/1606. 05250, 2016.
A. Aizawa, "An information-theoretic perspective of tf-idf measures, " Information Processing & Management, vol. 39, no. 1, pp. 45-65, 2003.
C. Manning, P. Raghavan, and H. Schutze, Introduction to Information Retrieval, 1st ed. Cambridge University Press, 2008.
E. Loper and S. Bird, "Nltk: The natural language toolkit, " in Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, 2002.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg et al., "Scikit-learn: Machine learning in Python, " Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, "Transformers: State-of-the-art natural language processing, " in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 2020.
T. Kluyver, B. Ragan-Kelley, F. Pérez, B. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. Hamrick, J. Grout, S. Corlay, P. Ivanov, D. Avila, S. Abdalla, and C. Willing, "Jupyter notebooks-a publishing format for reproducible computational workflows, " in Positioning and Power in Academic Publishing: Players, Agents and Agendas, 2016.
EU (GDPR), "Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), OJ L 119, 4. 5. 2016, p. 1-88, " 2016. [Online]. Available: Http://data. europa. eu/eli/reg/2016/679/oj
EU (2019/771), "Directive (EU) 2019/771 of the European Parliament and of the Council of 20 May 2019 on certain aspects concerning contracts for the sale of goods, amending Regulation (EU) 2017/2394 and Directive 2009/22/EC, and repealing Directive 1999/44/EC, OJ L 136, 22. 5. 2019, p. 28-50, " 2019. [Online]. Available: Http://data. europa. eu/eli/dir/2019/771/oj
"Law of 25 March 2020 (coordinated version) establishing a central electronic data retrieval system related to IBAN accounts and safe-deposit boxes, " 2020. [Online]. Available: Https://www. cssf. lu/en/Document/ law-of-25-march-2020-data-retrieval/
D. Harman, "Information retrieval evaluation, " Synthesis Lectures on Information Concepts, Retrieval, and Services, vol. 3, no. 2, pp. 1-119, 2011.
D. W. Zimmerman and B. D. Zumbo, "Relative power of the wilcoxon test, the friedman test, and repeated-measures anova on ranks, " The Journal of Experimental Education, vol. 62, no. 1, pp. 75-86, 1993.
G. J. Upton, "Fisher's exact test, " Journal of the Royal Statistical Society: Series A (Statistics in Society), vol. 155, no. 3, pp. 395-402, 1992.
J. Lin, Y. Liu, J. Guo, J. Cleland-Huang, W. Goss, W. Liu, S. Lohar, N. Monaikul, and A. Rasin, "TiQi: A natural language interface for querying software project data, " in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE, 2017, pp. 973-977.
N. Niu, W. Wang, A. Gupta, M. Assarandarban, L. Da Xu, J. Savolainen, and J.-R. C. Cheng, "Requirements sociotechnical graphs for managing practitioners' traceability questions, " IEEE Transactions on Computational Social Systems, vol. 5, no. 4, pp. 1152-1162, 2018.
C. Arora, M. Sabetzadeh, A. Goknil, L. Briand, and F. Zimmer, "NARCIA: An automated tool for change impact analysis in natural language requirements, " in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 962-965.
-, "Change impact analysis for natural language requirements: An NLP approach, " in 23rd IEEE International Requirements Engineering Conference, 2015.
G. M. Kanchev, P. K. Murukannaiah, A. K. Chopra, and P. Sawyer, "Canary: An interactive and query-based approach to extract requirements from online forums, " in 2017 IEEE 25th International Requirements Engineering Conference. IEEE, 2017, pp. 470-471.
E. M. Voorhees, "The TREC-8 question answering track report, " in Proceedings of The Eighth Text REtrieval Conference, vol. 500-246, 1999.
-, "Overview of the TREC 2001 question answering track, " in Proceedings of The Tenth Text REtrieval Conference, vol. 500-250, 2001.
S. M. Harabagiu, D. I. Moldovan, C. Clark, M. Bowden, A. Hickl, and P. Wang, "Employing two question answering systems in TREC 2005, " in Proceedings of the Fourteenth Text REtrieval Conference, vol. 500-266, 2005.
S. M. Harabagiu, M. Pasca, and S. J. Maiorano, "Opendomain textual question answering techniques, " Natural Language Engineering, vol. 9, pp. 231-267, 09 2003.
E. Grois, "Learning strategies for open-domain natural language question answering, " in 43rd Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. The Association for Computer Linguistics, 2005, pp. 85-90.
J. Lin, "The role of information retrieval in answering complex questions, " in 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. The Association for Computer Linguistics, 2006.
D. Demner-Fushman and J. Lin, "Answer extraction, semantic clustering, and extractive summarization for clinical question answering, " in 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. The Association for Computer Linguistics, 2006.
M. H. Heie, E. W. D. Whittaker, and S. Furui, "Optimizing question answering accuracy by maximizing log-likelihood, " in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. The Association for Computer Linguistics, 2010, pp. 236-240.
N. Duan, "Minimum bayes risk based answer re-ranking for question answering, " in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. The Association for Computer Linguistics, 2013, pp. 424-428.
Y. Yang, W.-t. Yih, and C. Meek, "WikiQA: A challenge dataset for open-domain question answering, " in Proceedings of the 2015 conference on empirical methods in natural language processing, 2015, pp. 2013-2018.
P. Rajpurkar, R. Jia, and P. Liang, "Know what you don't know: Unanswerable questions for SQuAD, " in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2018, pp. 784-789.
M. Hu, Y. Peng, F. Wei, Z. Huang, D. Li, N. Yang, and M. Zhou, "Attention-guided answer distillation for machine reading comprehension, " in Proceedings of Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018, pp. 2077-2086.
O. Khattab, C. Potts, and M. Zaharia, "Relevance-guided supervision for OpenQA with ColBERT, " Transactions of the Association for Computational Linguistics, vol. 9, pp. 929-944, 2021.
