Evaluating time-dependent methods and seasonal effects in code technical debt prediction

Robredo, Mikel; SAARIMÄKI, Nyyti; Esposito, Matteo; Taibi, Davide; Peñaloza, Rafael; Lenarduzzi, Valentina

doi:10.1016/j.jss.2025.112545

Download

Article (Scientific journals)

Evaluating time-dependent methods and seasonal effects in code technical debt prediction

Robredo, Mikel; SAARIMÄKI, Nyyti; Esposito, Matteo et al.

2025 • In Journal of Systems and Software, 230, p. 112545

Peer Reviewed verified by ORBi Dataset

Permalink
https://hdl.handle.net/10993/65529

DOI
10.1016/j.jss.2025.112545

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

2024_JSS_TDPrediction.pdf

Author postprint (859.7 kB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Technical debt; Software quality; Mining software repositories; Empirical software engineering; Time series analysis

Abstract :

[en] Background: Code Technical Debt (Code TD) prediction has gained significant attention in recent software engineering research. However, no standardized approach to Code TD prediction fully captures the factors influencing its evolution. Objective: Our study aims to assess the impact of time-dependent models and seasonal effects on Code TD prediction. It evaluates such models against widely used Machine Learning models also considering the influence of seasonality on prediction performance. Methods: We trained 11 prediction models with 31 Java open-source projects. To assess their performance, we predicted future observations of the SQALE index. To evaluate the practical usability of our TD forecasting model and their impact on practitioners, we surveyed 23 software engineering professionals. Results: Our study confirms the benefits of time-dependent techniques, with the ARIMAX model outperforming the others. Seasonal effects improved predictive performance, though the impact remained modest. ARIMAX/SARIMAX models demonstrated to provide well-balanced long-term forecasts. The survey highlighted strong industry interest in short- to medium-term TD forecasts. Conclusions: Our findings support using techniques that capture time dependence in historical software metric data, particularly for Code TD. Effectively addressing this evidence requires adopting methods that account for temporal patterns.

Research center :

Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation

Disciplines :

Computer science

Author, co-author :

Robredo, Mikel ; University of Oulu

SAARIMÄKI, Nyyti ; University of Luxembourg

Esposito, Matteo ; University of Oulu

Taibi, Davide; University of Oulu

Peñaloza, Rafael; University of Milano-Bicocca

Lenarduzzi, Valentina; University of Oulu

External co-authors :

yes

Language :

English

Title :

Evaluating time-dependent methods and seasonal effects in code technical debt prediction

Publication date :

December 2025

Journal title :

Journal of Systems and Software

ISSN :

0164-1212

eISSN :

1873-1228

Publisher :

Elsevier BV

Volume :

230

Pages :

112545

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://api.elsevier.com/content/article/PII:S0164121225002146?httpAccept=text/xml

Data Set :

Replication package for the Journal of Systems and Software (JSS) submitted article : "Evaluating Time-Dependent Methods and Seasonal Effects in Code Technical Debt Prediction"

Available on ORBilu :

since 08 August 2025

Statistics

Number of views

60 (5 by Unilu)

Number of downloads

39 (1 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Agresti, A., Foundations of Linear and Generalized Linear Models. 2015, John Wiley & Sons.
Akaike, H., A new look at the statistical model identification. IEEE Trans. Autom. Control 19 (1974), 716–723.
Allison, P.D., Missing data. The SAGE Handbook of Quantitative Methods in Psychology, 2009, 72–89.
Arcelli Fontana, F., Lenarduzzi, V., Roveda, R., Taibi, D., Are architectural smells independent from code smells? an empirical study. J. Syst. Softw. 154 (2019), 139–156.
Aversano, L., Bernardi, M., Cimitile, M., Iammarino, M., Montano, D., Forecasting technical debt evolution in software systems: an empirical study. Front. Comput. Sci., 17, 2022.
Basili, V., Caldiera, G., Rombach, D., The goal question metric approach. Encycl. Softw. Eng., 1994.
Blu, T., Thévenaz, P., Unser, M., Linear interpolation revitalized. IEEE Trans. Image Process. 13 (2004), 710–719.
Bordes, A., Bottou, L., Gallinari, P., Sgd-qn: Careful quasi-newton stochastic gradient descent. J. Mach. Learn. Res. 10 (2009), 1737–1754.
Box, G.E., Hillmer, S.C., Tiao, G.C., Analysis and modeling of seasonal time series. Seasonal Analysis of Economic Time Series, 1978, NBER, 309–344.
Box, G.E., Jenkins, G.M., Reinsel, G.C., Ljung, G.M., Time Series Analysis: Forecasting and Control. 2015, John Wiley & Sons.
Breiman, L., Random forests. Mach. Learn. 45 (2001), 5–32.
Brockwell, P.J., Davis, R.A., Time Series: Theory and Methods. 1991, Springer science & business media.
Brockwell, P.J., Davis, R.A., Calder, M.V., Introduction To Time Series and Forecasting, 2002, Springer.
Bruce, P., Bruce, A., Gedeck, P., Practical Statistics for Data Scientists: 50+ Essential Concepts using R and Python. 2020, O'Reilly Media.
Canova, F., Hansen, B.E., Are seasonal patterns constant over time? a test for seasonal stability. J. Bus. Econom. Statist. 13 (1995), 237–252.
Çarka, J., Esposito, M., Falessi, D., On effort-aware metrics for defect prediction. Empir. Softw. Eng., 27, 2022, 152.
Chan, K.-S., Cryer, J.D., Time Series Analysis with Applications in R. 2008, Springer.
Chen, T., Guestrin, C., 2016. Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining. pp. 785–794.
De Raedt, L., Flach, P., Machine Learning: ECML 2001: 12th European Conference on Machine Learning, Freiburg, Germany, September 5-7, 2001. Proceedings, 2003, Springer.
Dickey, D.A., Fuller, W.A., Distribution of the estimators for autoregressive time series with a unit root. J. Amer. Statist. Assoc. 74 (1979), 427–431.
Durbin, J., Koopman, S.J., Time Series Analysis By State Space Methods. 2012, Oxford University Press (UK).
Esposito, M., Falaschi, V., Falessi, D., An extensive comparison of static application security testing tools. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024, Salerno, Italy, June 18-21, 2024, 2024, ACM, 69–78.
Esposito, M., Falessi, D., Uncovering the hidden risks: The importance of predicting bugginess in untouched methods. 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation, SCAM, 2023, IEEE, 277–282.
Esposito, M., Moreschini, S., Lenarduzzi, V., Hästbacka, D., Falessi, D., Can we trust the default vulnerabilities severity?. L. Moonen, A. Gorla, (eds.) 23rd IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2023, Bogotá, Colombia, October 2-3, 2023, 2023, IEEE, 265–270.
Esposito, M., Palagiano, F., Leveraging large language models for preliminary security risk analysis: A mission-critical case study. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 2024, ACM, 442–445.
Esposito, M., Palagiano, F., Lenarduzzi, V., Taibi, D., On large language models in mission-critical it governance: Are we ready yet?. 2024 arXiv preprint arXiv:2412.11698.
Fida, M.A.F.A., Ahmad, T., Ntahobari, M., Variance threshold as early screening to boruta feature selection for intrusion detection system. 2021 13th International Conference on Information & Communication Technology and System, ICTS, 2021, IEEE, 46–50.
Flach, P., Machine Learning: The Art and Science of Algorithms that Make Sense of Data. 2012, Cambridge University Press.
for Computing Machinery, A., Acm publications policy on research involving human participants and subjects. 2021 https://www.acm.org/publications/policies/research-involving-human-participants-and-subjects.
Freund, R.J., Wilson, W.J., Sa, P., Regression Analysis. 2006, Elsevier.
Fuller, W.A., Introduction To Statistical Time Series. 2009, John Wiley & Sons.
Golub, G.H., Van Loan, C.F., Matrix Computations. 2013, JHU Press.
Hoerl, A.E., Kennard, R.W., Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12 (1970), 55–67.
Hyndman, R.J., Athanasopoulos, G., Forecasting: Principles and Practice. 2018, OTexts.
Hyndman, R.J., Khandakar, Y., Automatic time series forecasting: the forecast package for r. J. Stat. Softw. 27 (2008), 1–22.
James, G., Witten, D., Hastie, T., Tibshirani, R., et al. An Introduction To Statistical Learning, 2013, Springer.
Kim, J.H., Choi, I., Choosing the level of significance: A decision-theoretic approach. Abacus 57 (2021), 27–71.
Lenarduzzi, V., Saarimäki, N., Taibi, D., 2019a. The technical debt dataset. In: Conference on PREdictive Models and Data Analycs in Software Engineering. PROMISE ’19.
Lenarduzzi, V., Saarimäki, N., Taibi, D., 2019b. The technical debt dataset. In: Conference on Predictive Models and Data Analytics in Software Engineering.
Lenarduzzi, V., Saarimäki, N., Taibi, D., Some sonarqube issues have a significant but small effect on faults and changes. a large-scale empirical study. J. Syst. Softw., 2020, 110750.
Lenarduzzi, V., Sillitti, A., Taibi, D., 2017. Analyzing forty years of software maintenance models. In: International Conference on Software Engineering Companion. ICSE-C ’17, pp. 146–148.
Lenarduzzi, V., Sillitti, A., Taibi, D., A survey on code analysis tools for software maintenance prediction. International Conference in Software Engineering for Defence Applications, 2020, Springer International Publishing, 165–175.
Likert, R., A technique for the measurement of attitudes. Arch. Psychol., 1932.
Martin, F., Kent, B., Refactoring: Improving the Design of Existing Code. 1999, Addison-Wesley Longman Publishing Co. Inc.
Mathioudaki, M., Tsoukalas, D., Siavvas, M., Kehagias, D., Technical debt forecasting based on deep learning techniques. Computational Science and Its Applications – ICCSA 2021, 2021, 306–322.
Mathioudaki, M., Tsoukalas, D., Siavvas, M., Kehagias, D., Comparing univariate and multivariate time series models for technical debt forecasting. Computational Science and Its Applications, 2022, 62–78.
McQuarrie, A.D., Tsai, C.-L., Regression and Time Series Model Selection. 1998, World Scientific.
Mordal-Manet, K., Balmas, F., Denier, S., Ducasse, S., Wertz, H., Laval, J., Bellingard, F., Vaillergues, P., 2009. The squale model — A practice-based industrial quality model. In: International Conference on Software Maintenance. ICSME, pp. 531–534.
Olbrich, S., Cruzes, D.S., Basili, V., Zazworka, N., 2009. The evolution and impact of code smells: A case study of two open source systems. In: International Symposium on Empirical Software Engineering and Measurement. pp. 390–400.
Palomba, F., Bavota, G., Penta, M.D., Fasano, F., Oliveto, R., Lucia, A.D., On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigation. Empir. Softw. Eng. 23 (2018), 1188–1221.
Palomba, F., Tamburri, D.A., Arcelli Fontana, F., Oliveto, R., Zaidman, A., Serebrenik, A., Beyond technical aspects: How do community smells influence the intensity of code smells?. IEEE Trans. Softw. Eng., 2018 1–1.
Patton, M., Qualitative Evaluation and Research Methods. 2002, Sage.
Reichheld, F.F., The one number you need to grow. Harv. Bus. Rev. 81 (2003), 46–55.
Rios, N., Spínola, R.O., Mendonça, M., Seaman, C., The practitioners’ point of view on the concept of technical debt and its causes and consequences: a design for a global family of industrial surveys and its first results from Brazil. Empir. Softw. Eng. 25 (2020), 3216–3287.
Runeson, P., Höst, M., Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng. 14 (2009), 131–164.
Saarimäki, N., Moreschini, S., Lomio, F., Peñaloza, R., Lenarduzzi, V., 2022. Towards a robust approach to analyze time-dependent data in software engineering. In: International Conference on Software Analysis, Evolution and Reengineering. SANER, pp. 36–40.
Seaman, C., Guo, Y., Measuring and monitoring technical debt advances in computers. 2011.
Sharma, T., Singh, P., Spinellis, D., An empirical investigation on the relationship between design and architecture smells. Empir. Softw. Eng., 25, 2020.
Shumway, R.H., Stoffer, D.S., Stoffer, D.S., Time Series Analysis and Its Applications, 2000, Springer.
Sjoberg, D.I.K., Yamashita, A., Anda, B., Mockus, A., Dyba, T., Quantifying the effect of code smells on maintenance effort. IEEE Trans. Softw. Eng. 39 (2013), 1144–1156.
Spearman, C., The proof and measurement of association between two things. 1961.
Speiser, J.L., Miller, M.E., Tooze, J., Ip, E., A comparison of random forest variable selection methods for classification prediction modeling. Expert Syst. Appl. 134 (2019), 93–101.
Stone, M., Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B Stat. Methodol. 36 (1974), 111–133.
Straus, A., Techniques and procedures for developing grounded theory. Basics Qual. Res., 1998, 265–274.
Tibshirani, R., Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol., 58, 1996.
Tsoukalas, D., Kehagias, D., Siavvas, M., Chatzigeorgiou, A., Technical debt forecasting: An empirical study on open-source repositories. J. Syst. Softw., 170, 2020, 110777.
Tsoukalas, D., Mittas, N., Chatzigeorgiou, A., Kehagias, D., Ampatzoglou, A., Amanatidis, T., Angelis, L., Machine learning for technical debt identification. IEEE Trans. Softw. Eng. 48 (2021), 4892–4906.
Vassallo, C., Panichella, S., Palomba, F., Proksch, S., Gall, H.C., Zaidman, A., How developers engage with static analysis tools in different contexts. Empir. Softw. Eng., 2019.
Wang, X., Smith, K., Hyndman, R., Characteristic-based clustering for time series data. Data Min. Knowl. Discov. 13 (2006), 335–364.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A., Experimentation in Software Engineering: An Introduction. 2000, Springer Publishing Company, Incorporated.
Wohlin, C., Runeson, P., Höst, M., et al. Experimentation in Software Engineering. 2012, Springer.
Yu, L., Liu, H., 2003. Feature selection for high-dimensional data: A fast correlation-based filter solution. In: International Conference on Machine Learning. pp. 856–863.
Zozas, I., Bibi, S., Ampatzoglou, A., Forecasting the principal of code technical debt in javascript applications. IEEE Trans. Softw. Eng. 49 (2023), 2498–2512.