Tracing content requirements in financial documents using multi-granularity text analysis

[en] The completeness (in terms of content) of financial documents is a fundamental requirement for investment funds. To ensure completeness, financial regulators have to spend significant time carefully checking every financial document based on relevant content requirements, which prescribe the information types to be included in financial documents (e.g., the fund name, the description of shares’ issue conditions and procedures). Although several techniques have been proposed to automatically detect certain types of information in documents across application domains, they provide limited support to help regulators automatically identify the text chunks related to financial information types, due to the complexity of financial documents and the diversity of the sentences typically characterizing an information type. In this paper, we propose FITI to trace content requirements in financial documents with multi-granularity text analysis. Given a new financial document, FITI first selects a set of candidate sentences for efficient information type identification. Then, to rank candidate sentences, FITI uses a combination of rule-based and data-centric approaches, by leveraging information retrieval (IR) and machine learning (ML) techniques that analyze the words, sentences, and contexts related to an information type. Finally, using a list of domain-specific indicator phrases related to each information type, a heuristic-based selector, which considers both the sentence ranking and domain-specific phrases, determines a list of sentences corresponding to each information type. We evaluated FITI by assessing its effectiveness in tracing financial content requirements in 100 real-world financial documents. Experimental results show that FITI is able to provide accurate identification with average precision, recall, and F1-score values of 0.824, 0.646, and 0.716, respectively. The overall accuracy of FITI significantly outperforms the best baseline (based on a transformer language model) by 0.266 in terms of F1-score. Furthermore, FITI can help regulators detect about 80% of missing information types in financial documents.

Research center :

Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
NCER-FT - FinTech National Centre of Excellence in Research

Disciplines :

Computer science

Author, co-author :

Li, Xiaochen

BIANCULLI, Domenico ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV

Briand, Lionel

External co-authors :

yes

Language :

English

Title :

Tracing content requirements in financial documents using multi-granularity text analysis

Publication date :

April 2025

Journal title :

Requirements Engineering

ISSN :

0947-3602

eISSN :

1432-010X

Publisher :

Springer, Germany

Peer reviewed :

Peer Reviewed verified by ORBi

Focus Area :

Security, Reliability and Trust

FnR Project :

FNR16570468 - 2021 (01/03/2023-28/02/2025) - Gilbert Fridgen

Name of the research project :

U-AGR-8512 - NCER FT/ FinnovHub/ ICCOFIDO - BIANCULLI Domenico

Funders :

FNR - Fonds National de la Recherche

Funding number :

NCER22/IS/16570468/NCER-FT

Data Set :

FITI Experiment Results

Available on ORBilu :

since 22 November 2023

Statistics

Number of views

123 (18 by Unilu)

Number of downloads

38 (5 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Abreu C, Cardoso H, Oliveira E (2019) Findse@ fintoc-2019 shared task. In: Proc. Financial Narrative Processing Workshop (FNP), pp 69–73
Amaral O, Abualhaija S, Briand L (2023) Ml-based compliance verification of data processing agreements against gdpr. In: 2023 IEEE 31st International Requirements Engineering Conference (RE), IEEE, pp 53–64
Apache PDFBox project (2024) Pdfbox. https://pdfbox.apache.org
Arslan Y, Allix K, Veiber L, et al (2021) A comparison of pre-trained language models for multi-class text classification in the financial domain. In: WWW’21 Companion, pp 260–268
Azzopardi S, Colombo C, Pace GJ (2018) A controlled natural language for financial services compliance checking. In: Controlled Natural Language. MIT, p 11–20
G. Boella L.D. Caro L. Humphreys et al. Eunomos, a legal document and knowledge management system for the web to provide relevant, reliable and up-to-date information on the law Artif Intell Law 24 3 245 283 10.1007/s10506-016-9184-3
D. Bolchini P. Paolini Goal-driven requirements analysis for hypermedia-intensive web applications Requirements Eng 9 85 103 10.1007/s00766-004-0188-2
Bolchini D, Garzotto F, Paolini P (2007) Branding and communication goals for content-intensive interactive applications. In: 15th IEEE International Requirements Engineering Conference (RE 2007), IEEE, pp 173–182
Boutkova E, Houdek F (2011) Semi-automatic identification of features in requirement specifications. In: RE’11, IEEE, pp 313–318
Brito E, Sifa R, Bauckhage C, et al (2019) A hybrid ai tool to extract key performance indicators from financial reports for benchmarking. In: DocEng’19, pp 1–4
de Britto FA, Ferreira TC, Nunes LP, et al (2021) Comparing supervised machine learning techniques for genre analysis in software engineering research articles. In: Proc. Int’l Conf. on Recent Advances in Natural Language Processing (RANLP 2021), pp 63–72
T. Brown B. Mann N. Ryder et al. Language models are few-shot learners Adv Neural Inf Process Syst 33 1877 1901
J. Caramujo A.R. da Silva S. Monfared et al. Rsl-il4privacy: a domain-specific language for the rigorous specification of privacy policies Requirements Eng 24 1 1 26 10.1007/s00766-018-0305-2
S. Castano M. Falduti A. Ferrara et al. A knowledge-centered framework for exploration and retrieval of legal documents Inf Syst 106 10.1016/j.is.2021.101842 101842
Ceci M, Bianculli D, Briand LC (2024) Defining a model for content requirements from the law: An experience report. In: 32nd IEEE International Requirements Engineering Conference, RE 2024, Reykjavik, Iceland, June 24-28, 2024. IEEE, pp 18–30
Chalkidis I, Fergadiotis M, Manginas N, et al (2021) Regulatory compliance through Doc2Doc information retrieval: A case study in EU/UK legislation where text similarity has limitations. In: ACL’21, pp 3498–3511
Cleland-Huang J, Czauderna A, Gibiec M, et al (2010) A machine learning approach for tracing regulatory codes to product specific requirements. In: ICSE’10, pp 155–164
Cleland-Huang J, Gotel OC, Huffman Hayes J, et al (2014) Software traceability: trends and future directions. In: FOSE’14. ACM, p 55–69
P. Craja A. Kim S. Lessmann Deep learning for detecting financial statement fraud Decis Support Syst 139 10.1016/j.dss.2020.113421 113421
CSSF (The Commission de Surveillance du Secteur Financier) (2010) The Luxembourg law of 17 December 2010 relating to undertakings for collective investment. https://www.cssf.lu/wp-content/uploads/L_171210_UCI.pdf
J. Demšar Statistical comparisons of classifiers over multiple data sets J Mach Learn Res 7 1 30 2274360
Devlin J, Chang MW, Lee K, et al (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
Devlin J, Chang MW, Lee K, et al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL HLT’19, pp 4171–4186
Doyle D (2024) Default english stopwords. https://www.ranks.nl/stopwords
Fan M, Cheng D, Yang F, et al (2020) Fusing global domain information and local semantic information to classify financial documents. In: CIKM’20, pp 2413–2420
Feng F, Li M, Luo C, et al (2021) Hybrid learning to rank for financial event ranking. In: SIGIR’21, pp 233–243
Financial Times (2024) European ucits fund assets reach €10 tn landmark. https://www.ft.com/content/9551561c-0b60-3c2f-bb06-1610983d4ee5
J. Friedman T. Hastie R. Tibshirani et al. The elements of statistical learning 2 New York Springer Series in Statistics 587 588
R. Grishman Twenty-five years of information extraction Natural Language Eng 25 6 677 692 10.1017/S1351324919000512
J. Guo M. Gibiec J. Cleland-Huang Tackling the term-mismatch problem in automated trace retrieval Empirical Softw Eng 22 3 1103 1142 10.1007/s10664-016-9479-8
Guo Y, Korhonen A, Poibeau T (2011) A weakly-supervised approach to argumentative zoning of scientific documents. In: Empirical Methods in Natural language Processing (EMNLP). ACL, pp 273–283
Y. Guo R. Reichart A. Korhonen Unsupervised declarative knowledge induction for constraint-based learning of information structure in scientific documents Trans Assoc Comput Linguist(TACL) 3 131 143 10.1162/tacl_a_00128
Hayes JH, Dekhtyar A, Osborne J (2003) Improving requirements tracing via information retrieval. In: RE’03, IEEE, pp 138–147
J.H. Hayes A. Dekhtyar S.K. Sundaram Advancing candidate link generation for requirements tracing: The study of methods Trans Softw Eng 32 1 4 10.1109/TSE.2006.3
Huang HH, Chen HH (2017) Disa: A scientific writing advisor with deep information structure analysis. In: Proc. Int’l Joint Conf. on Artificial Intelligence (IJCAI), pp 5229–5231
Huang YJ, Lu J, Kurohashi S, et al (2019) Improving event coreference resolution by learning argument compatibility from unlabeled data. In: NAACL HLT’19, pp 785–795
Jiang J, Zhai C (2007) A systematic exploration of the feature space for relation extraction. In: NAACL HLT’07, pp 113–120
M. Joshi D. Chen Y. Liu et al. Spanbert: Improving pre-training by representing and predicting spans Trans Assoc Comput Linguist 8 64 77 10.1162/tacl_a_00300
Kumar A, Alam H, Werner T, et al (2016) Experiments in candidate phrase selection for financial named entity extraction-a demo. In: COLING’16 System Demonstrations, pp 45–48
Li H, Yang Q, Cao Y, et al (2020a) Cracking tabular presentation diversity for automatic cross-checking over numerical facts. In: SIGKDD’20, pp 2599–2607
Li J, Sun A, Han J, et al (2020b) A survey on deep learning for named entity recognition. Trans on Knowledge and Data Engineering
Li J, Bian Y, Wang G, et al (2023a) Cfgpt: Chinese financial assistant with large language model. arXiv:2309.10654
Li Q, Shah S, Fang R (2016) Table classification using both structure and content information: A case study of financial documents. In: Big Data’16, IEEE, pp 1778–1783
Q. Li Y. Chen J. Wang et al. Web media and stock markets: A survey and future directions from a big data perspective Trans on Knowledge Data Eng 30 2 381 399 10.1109/TKDE.2017.2763144
Li Y, Wang S, Ding H, et al (2023b) Large language models in finance: A survey. In: Proceedings of the Fourth ACM International Conference on AI in Finance. Association for Computing Machinery, New York, NY, USA, ICAIF ’23, p 374-382, https://doi.org/10.1145/3604237.3626869
C.D. Manning P. Raghavan H. Schütze Introduction to information retrieval Cambridge University Press 10.1017/CBO9780511809071
Merten T, Falis M, Hübner P, et al (2016) Software feature request detection in issue tracking systems. In: RE’16, IEEE, pp 166–175
W. Nuij V. Milea F. Hogenboom et al. An automated framework for incorporating news into stock trading strategies Trans on Knowledge Data Eng 26 4 823 835 10.1109/TKDE.2013.133
Porter M (2024) The porter stemming algorithm. https://tartarus.org/martin/PorterStemmer
Postolache O, Kruijff-Korbayová I, Kruijff GJM (2005) Data-driven approaches for information structure identification. In: Proc. Human Language Tech. Conf. and Conf. on Empirical Methods in Natural Language Processing (HLT/EMNLP), pp 9–16
P. Pullonen J. Tom R. Matulevičius et al. Privacy-enhanced bpmn: Enabling data privacy analysis in business processes models Softwa Syst Model 18 6 3235 3264 10.1007/s10270-019-00718-z
Rajpurkar P, Jia R, Liang P (2018) Know what you don’t know: Unanswerable questions for squad. In: ACL’18, pp 784–789
P. Ravisankar V. Ravi G.R. Rao et al. Detection of financial statement fraud and feature selection using data mining techniques Deci Support Syst 50 2 491 500 10.1016/j.dss.2010.11.006
Robaldo L, Caselli T, Russo I, et al (2011) From italian text to timeml document via dependency parsing. In: CICLing’11, Springer, pp 177–187
A. Rogers O. Kovaleva A. Rumshisky A primer in bertology: What we know about how bert works Trans Assoc Comput Linguist 8 842 866 10.1162/tacl_a_00349
Sannier N, Adedjouma M, Sabetzadeh M, et al (2017) Legal markup generation in the large: An experience report. In: Proc. Int’l Requirements Eng. Conf. (RE), IEEE, pp 302–311
C. Sansone G. Sperlí Legal information retrieval systems: State-of-the-art and open issues Inf Syst 106 10.1016/j.is.2021.101967 101967
Séaghdha DO, Teufel S (2014) Unsupervised learning of rhetorical structure with un-topic models. In: Proc. Int’l Conf. on Computational Linguistics (COLING), pp 2–13
L. Shi Z. Teng L. Wang et al. Deepclue: Visual interpretation of text-based deep stock prediction Trans on Knowledge Data Eng 31 6 1094 1108 10.1109/TKDE.2018.2854193
Stanford NLP Group (2024) Stanford nlp core. http://stanfordnlp.github.io
Torre D, Abualhaija S, Sabetzadeh M, et al (2020) An ai-assisted approach for checking the completeness of privacy policies against gdpr. In:, pp RE’20, IEEE136–146
Wang W, Dumont F, Niu N, et al (2020) Detecting software security vulnerabilities via requirements dependency analysis. Trans on Software Engineering
Y. Wang L. Shi M. Li et al. Detecting coreferent entities in natural language requirements Requirements Eng 27 3 351 373 10.1007/s00766-022-00374-8
Weischedel R, Pradhan S, Ramshaw L, et al (2011) Ontonotes release 4.0. LDC2011T03, Philadelphia, Penn: Linguistic Data Consortium
Wu S, Irsoy O, Lu S, et al (2023) Bloomberggpt: A large language model for finance. arXiv:2303.17564
J. Yang H. Jin R. Tang et al. Harnessing the power of llms in practice: A survey on chatgpt and beyond ACM Trans Knowl Discov Data 10.1145/3649506 just Accepted
Zhang J, El-Gohary N (2011) Automated information extraction from construction-related regulatory documents for automated compliance checking. In: Proceedings of the 2011 CIB World Congress
J. Zhang N. El-Gohary Semantic nlp-based information extraction from construction regulatory documents for automated compliance checking J Comput Civil Eng 30 2 04015014 10.1061/(ASCE)CP.1943-5487.0000346
R. Zhang N. El-Gohary A deep neural network-based method for deep information extraction using transfer learning strategies to support automated compliance checking Autom Constr 132 10.1016/j.autcon.2021.103834 103834
Zhang T, Wu F, Katiyar A, et al (2021) Revisiting few-sample bert fine-tuning. In: ICLR’21, to appear
Zhang Y, Zhong V, Chen D, et al (2017) Position-aware attention and supervised data improve slot filling. In: EMNLP’17, pp 35–45
Zhang Y, Zhao P, Li B, et al (2020) Cost-sensitive portfolio selection via deep reinforcement learning. Trans on Knowledge and Data Engineering