Fine-tuning GPT-3 for legal rule classification

LIGA, Davide; ROBALDO, Livio

doi:10.1016/j.clsr.2023.105864

No full text

Article (Scientific journals)

Fine-tuning GPT-3 for legal rule classification

LIGA, Davide; ROBALDO, Livio

2023 • In Computer Law & Security Review, 51, p. 105864

Editorial reviewed

Permalink
https://hdl.handle.net/10993/57759

DOI
10.1016/j.clsr.2023.105864

Files (0)Send to Details Statistics Bibliography Similar publications

Files

Full Text

No document available.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

AI&Law; GPT-3; Rule classification; AI&law; Classification tasks; Fine tuning; Language model; Legal domains; Legal rules; XML standards; Law

Abstract :

[en] In this paper, we propose a Legal Rule Classification (LRC) task using one of the most discussed language model in the field of Artificial Intelligence, namely GPT-3, a generative pretrained language model. We train and test the proposed LRC task on the GDPR encoded in LegalDocML (Palmirani and Vitali, 2011) and LegalRuleML (Athan et al., 2013), two widely used XML standards for the legal domain. We use the LegalDocML and LegalRuleML annotations provided in Robaldo et al. (2020) to fine-tuned GPT-3. While showing the ability of large language models (LLMs) to easily learn to classify legal and deontic rules even on small amount of data, we show that GPT-3 can significantly outperform previous experiments on the same task. Our work focused on a multiclass task, showing that GPT-3 is capable to recognize the difference between obligation rules, permission rules and constitutive rules with performances that overcome previous scores in LRC.

Disciplines :

Computer science

Author, co-author :

LIGA, Davide ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

ROBALDO, Livio ; Legal Innovation Lab Wales, Swansea University, Swansea, United Kingdom

External co-authors :

yes

Language :

English

Title :

Fine-tuning GPT-3 for legal rule classification

Publication date :

November 2023

Journal title :

Computer Law & Security Review

ISSN :

2212-473X

eISSN :

2212-4748

Publisher :

Elsevier Ltd

Volume :

Pages :

105864

Peer reviewed :

Editorial reviewed

Additional URL :

https://api.elsevier.com/content/article/PII:S0267364923000742?httpAccept=text/xml

Funding text :

Livio Robaldo has been supported by the Legal Innovation Lab Wales operation within Swansea University's Hillary Rodham Clinton School of Law. The operation has been part-funded by the European Regional Development Fund through the Welsh Government.Davide Liga was supported by the project INDIGO, which is financially supported by the NORFACE Joint Research Programme on Democratic Governance in a Turbulent Age and co-funded by AEI, AKA, DFG and FNR and the European Commission through Horizon 2020 under grant agreement No 822166 .

Available on ORBilu :

since 23 November 2023

Statistics

Number of views

126 (6 by Unilu)

Number of downloads

0 (0 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

WoS citations^™

Bibliography

Ashley, K.D., Artificial intelligence and legal analytics: new tools for law practice in the digital age. 2017, Cambridge University Press.
Athan, T., Boley, H., Governatori, G., Palmirani, M., Paschke, A., Wyner, A., Oasis LegalRuleML. Proceedings of the fourteenth international conference on artificial intelligence and law, 2013, 3–12.
Boella, G., Caro, L.D., Rispoli, D., Robaldo, L., A system for classifying multi-label text into EuroVoc. ICAIL, 2013, ACM, 239–240.
Boella, G., Di Caro, L., Humphreys, L., Robaldo, L., Rossi, R., van der Torre, L., Eunomos, a legal document and knowledge management system for the web to provide relevant, reliable and up-to-date information on the law. Artif Intell Law, 4, 2016.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D., Language models are few-shot learners. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H., (eds.) Advances in neural information processing systems, 2020, Curran Associates, Inc., 1877–1901 Vol. 33URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
Chalkidis I., Androutsopoulos I., Michos A. Obligation and prohibition extraction using hierarchical RNNS, arXiv preprint arXiv:1805.03871.
Chalkidis I., Fergadiotis M., Malakasiotis P., Aletras N., Androutsopoulos I. Legal-bert: the muppets straight out of law school, arXiv preprint arXiv:2010.02559.
Conover M., Hayes M., Mathur A., Xie J., Wan J., Shah S., Ghodsi A., Wendell P., Zaharia M., Xin R. Free dolly: introducing the world's first truly open instruction-tuned llm (2023). URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
de Maat, E., Winkels, R., Automatic classification of sentences in Dutch laws. Legal knowledge and information systems, 2008, IOS Press, 207–216.
Devlin J., Chang M.-W., Lee K., Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805.
Dragoni, M., Villata, S., Rizzi, W., Governatori, G., Combining natural language processing approaches for rule extraction from legal documents. Pagallo, U., Palmirani, M., Casanovas, P., Sartor, G., Villata, S., (eds.) AI approaches to the complexity of legal systems, 2018, Springer International Publishing, Cham, 287–300.
Gao, X., Singh, M.P., Extracting normative relationships from business contracts. Proceedings of the 2014 international conference on autonomous agents and multi-agent systems, 2014, 101–108.
Gomez-Perez, J.M., Denaux, R., Garcia-Silva, A., Hybrid natural language processing: an introduction. 2020, Springer International Publishing, Cham, 3–6, 10.1007/978-3-030-44830-1_1 doi:10.1007/978-3-030-44830-1_1. URL.
Humphreys, L., Boella, G., van der Torre, L., Robaldo, L., Di Caro, L., Ghanavati, S., Muthuri, R., Populating legal ontologies using semantic role labeling. Artif Intell Law 29 (2021), 171–211.
Joshi, V., Anish, P.R., Ghaisas, S., Domain adaptation for an automated classification of deontic modalities in software engineering contracts. Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, 2021, 1275–1280.
Kiyavitskaya, N., Zeni, N., Breaux, T.D., Anto´n, A.I., Cordy, J.R., Mich, L., Mylopoulos, J., Automating the extraction of rights and obligations for regulatory compliance. International conference on conceptual modeling, 2008, Springer, 154–168.
Liga, D., Palmirani, M., Transfer learning for deontic rule classification: the case study of the GDPR. International conference on legal knowledge and information systems, 2022, EasyChair, 2022 Saarbru¨cken 14-16 December.
Liga, D., Palmirani, M., Deontic sentence classification using tree kernel classifiers. Intelligent systems and applications: proceedings of the 2022 intelligent systems conference (IntelliSys), 2022, Springer International Publishing Cham, 54–73 Volume 1.
Liga, D., Palmirani, M., Transfer learning with sentence embeddings for argumentative evidence classification. Proceedings of the 20th workshop on computational models of natural argument, 2020.
Liga D. Hybrid artificial intelligence to extract patterns and rules from argumentative and legal texts.
Neill, J.O., Buitelaar, P., Robin, C., Brien, L.O., Classifying sentential modality in legal language: a use case in financial regulations, acts and directives. Proceedings of the 16th edition of the international conference on artificial intelligence and law, 2017, 159–168.
Nguyen, M., Nguyen, T., Tran, V., Nguyen, H., Nguyen, L., Satoh, K., Learning to map the GDPR to logic representation on DAPRECO-KB. ACIIDS, 2022, Springer, 442–454 (1), Vol. 13757 of Lecture Notes in Computer Science.
Palmirani, M., Vitali, F., Akoma-Ntoso for legal documents. Legislative XML for the semantic web, 2011, Springer, 75–100.
Radford A., Narasimhan K. Improving language understanding by generative pre-training, 2018.
Robaldo, L., Villata, S., Wyner, A., Grabmair, M., Introduction for artificial intelligence and law: special issue ”natural language processing for legal texts. Artif Intell Law 27:2 (2019), 113–115.
Robaldo, L., Bartolini, C., Palmirani, M., Rossi, A., Martoni, M., Lenzini, G., Formalizing gdpr provisions in reified i/o logic: the DAPRECO knowledge base. J Logic Lang Inform 29:4 (2020), 401–449.
Robaldo, L., Bartolini, C., Lenzini, G., The dapreco knowledge base: representing the GDPR in LegalRuleML. Proceedings of the 12th language resources and evaluation conference, 2020, 5688–5697.
Robaldo, L., Towards compliance checking in reified I/O logic via SHACL. Maranha˜o, J., Wyner, A.Z., (eds.) Proc. of 18th international conference for artificial intelligence and law (ICAIL), 2021, ACM.
Rodr´ıguez-Doncel, V., Palmirani, M., Araszkiewicz, M., Casanovas, P., Pagallo, U., Sartor, G., Introduction: A hybrid regulatory framework and technical architecture for a human-centered and explainable AI, in: AI Approaches to the Complexity of Legal Systems XI-XII. 2020, Springer, 1–11.
Sanh V., Debut L., Chaumond J., Wolf T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108.
Scao T.L., Fan A., Akiki C., Pavlick E., Ilic´ S., Hesslow D., Castagne´ R., Luccioni A.S., Yvon F., Galle´ M., et al. Bloom: a 176b-parameter open-access multilingual language model, arXiv preprint arXiv:2211.05100.
Shaghaghian, S., Feng, L.Y., Jafarpour, B., Pogrebnyakov, N., Customizing contextualized language models for legal document reviews. 2020 IEEE international conference on big data (Big Data), 2020, IEEE, 2139–2148.
Song, D., Vold, A., Madan, K., Schilder, F., Multi-label legal document classification: a deep learning-based approach with label- attention and domain-specific pre-training. Inf Syst, 106, 2022.
Sun, X., Robaldo, L., On the complexity of input/output logic. J Appl Logic 25 (2017), 69–88.
Team M.N. Introducing mpt-7b: a new standard for open-source, commercially usable llms, accessed: 2023-05-05 (2023). URL www.mosaicml.com/blog/mpt-7b.
Touvron H., Lavril T., Izacard G., Martinet X., Lachaux M.-A., Lacroix T., Rozie`re B., Goyal N., Hambro E., Azhar F., et al. Llama: open and efficient foundation language models, arXiv preprint arXiv:2302.13971.
Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., Kaiser L., Polosukhin I. Attention is all you need, 2017. URL https://arxiv.org/pdf/1706.03762.pdf.
Waltl, B., Muhr, J., Glaser, I., Bonczek, G., Scepankova, E., Matthes, F., Classifying legal norms with active machine learning. JURIX, 2017, 11–20.
Wyner, A., Peters, W., On rule extraction from regulations. Legal knowledge and information systems, 2011, IOS Press, 113–122.