A Multi-solution Study on GDPR AI-enabled Completeness Checking of DPAs

Specifying legal requirements for software systems to ensure their
compliance with the applicable regulations is a major concern of requirements
engineering. Personal data which is collected by an organization is often shared
with other organizations to perform certain processing activities. In such cases,
the General Data Protection Regulation (GDPR) requires issuing a data processing
agreement (DPA) which regulates the processing and further ensures
that personal data remains protected. Violating GDPR can lead to huge fines
reaching to billions of Euros. Software systems involving personal data processing
must adhere to the legal obligations stipulated both at a general level
in GDPR as well as the obligations outlined in DPAs highlighting specific business.
In other words, a DPA is yet another source from which requirements
engineers can elicit legal requirements. However, the DPA must be complete
according to GDPR to ensure that the elicited requirements cover the complete
set of obligations. Therefore, checking the completeness of DPAs is a
prerequisite step towards developing a compliant system. Analyzing DPAs
with respect to GDPR entirely manually is time consuming and requires adequate
legal expertise. In this paper, we propose an automation strategy that
addresses the completeness checking of DPAs against GDPR provisions as a
text classification problem. Specifically, we pursue ten alternative solutions
which are enabled by different technologies, namely traditional machine learning,
deep learning, language modeling, and few-shot learning. The goal of our
work is to empirically examine how these different technologies fare in the legal
domain. We computed F2 score on a set of 30 real DPAs. Our evaluation shows
that best-performing solutions yield F2 score of 86.7% and 89.7% are based on pre-trained BERT and RoBERTa language models. Our analysis further
shows that other alternative solutions based on deep learning (e.g., BiLSTM)
and few-shot learning (e.g., SetFit) can achieve comparable accuracy, yet are
more efficient to develop.