[en] [Context] Microservices enable the decomposition of applications into small, independent, and connected services. The independence between services could positively affect a project’s velocity, which is considered an important maintenance metric measuring the time taken to implement features and fix bugs. However, no studies have investigated the causal relationship between microservices and velocity. [Objective and Method] The goal of this study is to investigate the effect of microservices on velocity which is a common maintenance metric. The study compares projects on GitHub developed with microservices style from the beginning and similar projects using monolithic architectures. The study was conducted as a retrospective cohort study, which is a study type used to assess causality. [Results] The results did not find statistically significant differences in mean velocities in microservice-based and monolithic projects. Furthermore, the statistical adjustment performed to quantify the statistical impact of the use of microservices on velocity considering additional confounders did not find statistically significant impact from these. [Conclusions] The results did not indicate a difference between microservices-based projects and monolithic projects in terms of velocity. In addition, this study will contribute to the body of knowledge of empirical methods and be among the first works to adopt the methodology of the cohort study.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation Empirical Software Engineering in Software, Systems and Services, University of Oulu Facultad de Informática, Universidad Politécnica de Madrid
Disciplines :
Computer science
Author, co-author :
SAARIMÄKI, Nyyti ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV
Robredo, Mikel; University of Oulu
Lenarduzzi, Valentina ; University of Oulu
Vegas, Sira; Universidad Politécnica de Madrid
Juristo, Natalia; Universidad Politécnica de Madrid
Taibi, Davide; University of Oulu
External co-authors :
yes
Language :
English
Title :
Does microservice adoption impact the velocity? A cohort study
University of Oulu FNR - Luxembourg National Research Fund ERDF - European Regional Development Fund AEI - Agencia Estatal de Investigación
Funding number :
PID2022-137846NB-I00; C22/IS/17373407/LOGODOR
Funding text :
Open Access funding provided by University of Oulu (including Oulu University Hospital). This
research was supported by grant PID2022-137846NB-I00 funded by MCIN/AEI/10.
13039/501100011033 and by “ERDF A way of making Europe”.
This research was funded in whole, or in part, by the Luxembourg National Research Fund (FNR), grant
reference C22/IS/17373407/LOGODOR. For the purpose of open access, and in fulfillment of the obligations
arising from the grant agreement, the authors have applied a Creative Commons Attribution 4.0 International
(CC BY 4.0) license to any Author Accepted Manuscript Version arising from this submission.
A. Agresti Foundations of linear and generalized linear models John Wiley & Sons
Allen R (2017) Statistics and experimental design for psychologists: A model comparison approach. World Scientific Publishing Company
D. Amoroso d’Aragona A. Bakhtin X. Li R. Su L. Adams E. Aponte F. Boyle P. Boyle R. Koerner J. Lee F. Tian Y. Wang J. Nyyssölä E. Quevedo S. Md Rahaman A.S. Abdelfattah M. Mäntylä T. Cerny D. Taibi A dataset of microservices-based open-source projects International conference on mining software repositories (MSR) 21 504 509 10.1145/3643991.3644890
P.C. Austin Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples Stat Med 28 25 3083 3107 2750408 10.1002/sim.3697
P.C. Austin E.A. Stuart Moving towards best practice when using inverse probability of treatment weighting (iptw) using the propensity score to estimate causal treatment effects in observational studies Stat Med 34 28 3661 3679 3422140 10.1002/sim.6607
H.M. Ayas P. Leitner R. Hebig An empirical study of the systemic and technical migration towards microservices Empir Softw Eng 28 4 85 10.1007/S10664-023-10308-9
K. Benson A.J. Hartz A comparison of observational studies and randomized, controlled trials N Engl J Med 342 25 1878 1886 10.1056/NEJM200006223422506
M. Bianchi M. Boyle D. Hollingsworth A comparison of methods for trend estimation Appl Econ Lett 6 2 103 109 10.1080/135048599353726
R.C. Blair J.J. Higgins A comparison of the power of wilcoxon’s rank-sum statistic to that of student’s t statistic under various nonnormal distributions J Educ Stat 5 4 309 335
G.E. Box D.R. Cox An analysis of transformations J Royal Stat Soc Ser B Stat Methodol 26 2 211 243 10.1111/j.2517-6161.1964.tb00553.x
Boyter B (2018) Sloc cloc and code (scc). https://github.com/boyter/scc
R.M. Carroll L.A. Nordholm Sampling characteristics of kelley’s ε and hays’ ω Educ Psychol Meas 35 3 541 554 10.1177/001316447503500304
G. Casella R. Berger Statistical inference CRC Press 10.1201/9781003456285
W.W. Chin et al. The partial least squares approach to structural equation modeling Mod Methods Bus Res 295 2 295 336
C. Cinelli C. Hazlett Making sense of sensitivity: Extending omitted variable bias J Royal Stat Soc Ser B Stat Methodol 82 1 39 67 4060976 10.1111/rssb.12348
J. Cohen Statistical power analysis for the behavioral sciences Routledge 10.4324/9780203771587
J. Concato N. Shah R.I. Horwitz Randomized, controlled trials, observational studies, and the hierarchy of research designs N Engl J Med 342 25 1887 1892 10.1056/NEJM200006223422507
M.E. Di Cicco V. Ragazzo T. Jacinto Mortality in relation to smoking: the british doctors study Breathe 12 275 276 10.1183/20734735.013416
Dragoni N, Giallorenzo S, Lafuente AL, Mazzara M, Montesi F, Mustafin R, Safina L (2017) Microservices: yesterday, today, and tomorrow. In: Present and ulterior software engineering, Springer, pp 195–216
Dragoni N, Lanese I, Larsen ST, Mazzara M, Mustafin R, Safina L (2018) Microservices: How to make your application scale. In: Petrenko AK, Voronkov A (eds) Perspectives of system informatics, springer international publishing, Cham, pp 95–104
Du H, Zhang Z, Yuan KH (2016) (2017) Power analysis for t-test with non-normal data and unequal variances. Quantitative psychology: The 81st annual meeting of the psychometric society, Asheville, North Carolina. Springer, pp 373–380
P.D. Ellis The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results Cambridge University Press 10.1017/CBO9780511761676
T. Faresjö Å. Faresjö To match or not to match in epidemiological studies–same outcome but less power Int J Environ Res Public Health 7 1 325 332 10.3390/ijerph7010325
Field A (2024) Discovering statistics using IBM SPSS statistics. Sage publications limited
Fowler M, Lewis J (2014) Microservices a definition of this new architectural term. https://bit.ly/3zk5xXr
L. Gordis Epidemiology, fifth, edition Philadelphia Elsevier Saunders
D.A. Grimes K.F. Schulz Descriptive studies: what they can and cannot do Lancet 359 9301 145 149 10.1016/S0140-6736(02)07373-7
A. Hart Mann-whitney test is not just a test of medians: differences in spread can be important Bmj 323 7309 391 393 10.1136/bmj.323.7309.391
D.R. Helsel R.M. Hirsch Statistical methods in water resources Elsevier
D.E. Ho K. Imai G. King E.A. Stuart Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference Polit Anal 15 3 199 236 10.1093/pan/mpl013
M. Hollander D.A. Wolfe E. Chicken Nonparametric statistical methods John Wiley & Sons
Inc D (2023) Dockerfile reference. https://docs.docker.com/engine/reference/builder/
P. Jamshidi C. Pahl N.C. Mendonça J. Lewis S. Tilkov Microservices: The journey so far and challenges ahead IEEE Software 35 3 24 35 10.1109/MS.2018.2141039
Jangla K (2018a) Accelerating Development Velocity Using Docker: Docker Across Microservices. apress
Jangla K (2018b) Accelerating Development Velocity Using Docker: Docker Across Microservices. Apress
V. Lenarduzzi F. Lomio N. Saarimäki D. Taibi Does migrating a monolithic system to microservices decrease the technical debt? J Syst Softw 169 10.1016/j.jss.2020.110710 110710
Lomio F, Codabux Z, Birtch D, Hopkins D, Taibi D (2022) On the benefits of the accelerate metrics: An industrial survey at vendasta. In: 2022 IEEE international conference on software analysis, evolution and reengineering (SANER), pp 46–50, https://doi.org/10.1109/SANER53432.2022.00017
Y. Ma T. Dey C. Bogart S. Amreen M. Valiev A. Tutko D. Kennard R. Zaretzki A. Mockus World of code: enabling a research workflow for mining and analyzing the universe of open source vcs data Empir Softw Eng 26 1 42 10.1007/s10664-020-09905-9
Mann HB (1945) Nonparametric tests against trend. Econometrica: Journal of the econometric society, pp 245–259
Moore DS, McCabe GP (1989) Introduction to the practice of statistics. WH Freeman/Times Books/Henry Holt & Co
Morabia A (2004) A history of epidemiologic methods and concepts. vol 1
Newman S (2015) Building Microservices: Designing Fine-Grained Systems, 1st edn. O’Reilly Media
Newman S (2021) Building microservices: designing fine-grained systems. " O’Reilly Media, Inc."
O’brien RM (2007) A caution regarding rules of thumb for variance inflation factors. Quality & quantity 41:673–690
S. Olejnik J. Algina Generalized eta and omega squared statistics: measures of effect size for some common research designs Psychol Methods 8 4 434 10.1037/1082-989X.8.4.434
Oppewal H (2010) Concept of causality and conditions for causality. Wiley International encyclopedia of marketing
Pahl C, Jamshidi P, Zimmermann O (2018) Architectural principles for cloud software. ACM Trans Internet Technol 18(2):Article No.: 17
N.M. Razali Y.B. Wah et al. Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests J Stat Model Anal 2 1 21 33
Rosenbaum P (2002) Observational studies, 2nd edn springer. New York, New York, USA[Google Scholar]
Rosenbaum PR (2010) Design of Observational Studies, 1st edn. Springer Series in Statistics, Springer-Verlag, New York, NY
J.P. Royston An extension of shapiro and wilk’s w test for normality to large samples J Royal Stat Soc Ser C (Appl Stat) 31 2 115 124
D.B. Rubin Using propensity scores to help design observational studies: application to the tobacco litigation Health Serv Outcomes Res Methodol 2 169 188 10.1023/A:1020363010465
Saarimäki N, Lenarduzzi V, Vegas S, Juristo N, Taibi D (2020) Cohort studies in software engineering: A vision of the future. In: Proceedings of the 14th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM), pp 1–6
Saarimäki N, Manero MR, Juristo N, Taibi D, Lenarduzzi V, et al. (2023) Does microservices adoption impact the development velocity? a cohort study. a registered report. arXiv preprint arXiv:2306.02034
Sheather S (2009) A modern approach to regression with R. Springer Science & Business Media
J. Soldani D.A. Tamburri W.J.V.D. Heuvel The pains and gains of microservices: A systematic grey literature review J Syst Softw 146 215 232 10.1016/j.jss.2018.09.082
J.W. Song K.C. Chung Observational studies: cohort and case-control studies Plast Reconstr Surg 126 6 2234 10.1097/PRS.0b013e3181f44abc
Stuart EA, King G, Imai K, Ho D (2011) Matchit: nonparametric preprocessing for parametric causal inference. J Stat Softw
E.A. Stuart B.K. Lee F.P. Leacy Prognostic score-based balance measures can be a useful diagnostic for propensity score methods in comparative effectiveness research J clin Epidemiol 66 8 S84 S90 10.1016/j.jclinepi.2013.01.013
D. Taibi V. Lenarduzzi C. Pahl Processes, motivations, and issues for migrating to microservices architectures: An empirical investigation IEEE Cloud Comput 4 5 22 32 10.1109/MCC.2017.4250931
Trichopoulos D, Kalandidi A, Sparros L, Macmahon B (1981) Lung cancer and passive smoking. Int J Cancer 27(1):1–4. https://doi.org/10.1002/ijc.2910270102
T.J. VanderWeele Principles of confounder selection Eur J Epidemiol 34 211 219 10.1007/s10654-019-00494-6
E. Von Elm D.G. Altman M. Egger S.J. Pocock P.C. Gøtzsche J.P. Vandenbroucke The strengthening the reporting of observational studies in epidemiology (strobe) statement: guidelines for reporting observational studies Lancet 370 9596 1453 1457 10.1016/S0140-6736(07)61602-X
Waseem M, Liang P, Márquez G, Salle AD (2020) Testing microservices architecture-based applications: A systematic mapping study. In: 2020 27th asia-pacific software engineering conference (APSEC), pp 119–128, https://doi.org/10.1109/APSEC51365.2020.00020
Wu M, Zhang Y, Liu J, Wang S, Zhang Z, Xia§ X, Mao X (2022) On the way to microservices: Exploring problems and solutions from online q &a community. In: 2022 IEEE international conference on software analysis, evolution and reengineering (SANER), pp 432–443, https://doi.org/10.1109/SANER53432.2022.00058
Zhang Z (2014) Monte carlo based statistical power analysis for mediation models: Methods and software. Behav Res Methods 46:1184–1198
