[en] Modern software development is a complex engineering process where developer code cohabits with an increasingly larger number of external open-source components. Even though these components facilitate sharing and reusing code along with other benefits related to maintenance and code quality, they are often the seeds of vulnerabilities in the software supply chain leading to attacks with severe consequences. Indeed, one common strategy used to conduct attacks is to exploit or inject other security flaws in new versions of dependency packages. It is thus important to keep dependencies updated in a software development project. Unfortunately, several prior studies have highlighted that, to a large extent, developers struggle to keep track of the dependency package updates, and do not quickly incorporate security patches. Therefore, automated dependency-update bots have been proposed to mitigate the impact and the emergence of vulnerabilities in open-source projects. In our study, we focus on Dependabot, a dependency management bot that has gained popularity on GitHub recently. It allows developers to keep a lookout on project dependencies and reduce the effort of monitoring the safety of the software supply chain. We performed a large empirical study on dependency updates and security pull requests to understand: (1) the degree and reasons of Dependabot’s popularity; (2) the patterns of developers’ practices and techniques to deal with vulnerabilities in dependencies; (3) the management of security pull requests (PRs), the threat lifetime, and the fix delay; and (4) the factors that significantly correlate with the acceptance of security PRs and fast merges. To that end, we collected a dataset of 9,916,318 pull request-related issues made in 1,743,035 projects on GitHub for more than 10 different programming languages. In addition to the comprehensive quantitative analysis, we performed a manual qualitative analysis on a representative sample of the dataset, and we substantiated our findings by sending a survey to developers that use dependency management tools. Our study shows that Dependabot dominates more than 65% of dependency management activity, mainly due to its efficiency, accessibility, adaptivity, and availability of support. We also found that developers handle dependency vulnerabilities differently, but mainly rely on the automation of PRs generation to upgrade vulnerable dependencies. Interestingly, Dependabot’s and developers’ security PRs are highly accepted, and the automation allows to accelerate their management, so that fixes are applied in less than one day. However, the threat of dependency vulnerabilities remains hidden for 512 days on average, and patches are disclosed after 362 days due to the reliance on the manual effort of security experts. Also, project characteristics, the amount of PR changes, as well as developer and dependency features seem to be highly correlated with the acceptance and fast merges of security PRs.
Centre de recherche :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > TruX - Trustworthy Software Engineering
Disciplines :
Sciences informatiques
Auteur, co-auteur :
Rebatchi, Hocine ; École de Technologie Supérieure, Montreal, Canada
Moha, Naouel; École de Technologie Supérieure, Montreal, Canada
This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), and the European Research Council (ERC) under the European Union\u2019s Horizon 2020 research and innovation program (grant agreement No. 949014).This study was supported by (1) the Natural Sciences and Engineering Research Council of Canada (NSERC), and by (2) the European Research Council (ERC) under the European Union\u2019s Horizon 2020 research and innovation program (grant agreement No. 949014).
Akoglu H (2018) User’s guide to correlation coefficients. Turk J Emerg Med 18(3):91–93. https://doi.org/10.1016/j.tjem.2018.08.001, https://www.sciencedirect.com/science/article/pii/S2452247318302164
Alfadel M, Costa DE, Shihab E, Mkhallalati M (2021) On the use of dependabot security pull requests. In: 2021 IEEE/ACM 18th International conference on mining software repositories (MSR), pp 254–265. https://doi.org/10.1109/MSR52588.2021.00037
A. Andreoli A. Lounis M. Debbabi A. Hanna On the prevalence of software supply chain attacks: empirical study and investigative framework Forensic Sci Int: Digital Investigation 2023 44 301508
Angermeir F, Voggenreiter M, Moyón F, Mendez D (2021) Enterprise-driven open source software: a case study on security automation. In: 2021 IEEE/ACM 43rd International conference on software engineering: software engineering in practice (ICSE-SEIP). IEEE, pp 278–287
ben Othmane L, Chehrazi G, Bodden E, Tsalovski P, Brucker AD, Miseldine P (2015) Factors impacting the effort required to fix security vulnerabilities. In: Lopez J, Mitchell CJ (eds) Information security. Springer International Publishing, Cham, pp 102–119
L. Ben Othmane G. Chehrazi E. Bodden P. Tsalovski A.D. Brucker Time for addressing software security issues: prediction models and impacting factors Data Sci Eng 2017 2 2 107 124 10.1007/s41019-016-0019-8
Bird C, Gourley A, Devanbu P, Swaminathan A, Hsu G (2007) Open borders? immigration in open source projects. In: Proceedings of the fourth international workshop on mining software repositories. IEEE Computer Society, USA, MSR ’07, p 6. https://doi.org/10.1109/MSR.2007.23
Birsan A (2021) Dependency confusion: how I hacked into apple, microsoft and dozens of other companies. https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
Boehm C (2023) Supply chain attacks: how to protect against attack and sabotage. https://assets.sentinelone.com/supply-chain-attacks/how-to-protect-against-attack-and-sabotage-en
Calkins KG (2005) Correlation coefficients. https://www.andrews.edu/~calkins/math/edrm611/edrm05.htm, publisher: Andrews University
G. Canfora A. Di Sorbo S. Forootani A. Pirozzi C.A. Visaggio Investigating the vulnerability fixing process in oss projects: peculiarities and challenges Comput Secur 2020 99 102067 10.1016/j.cose.2020.102067
Coufalíková A, Klaban I, Šlajs T (2021) Complex strategy against supply chain attacks. In: 2021 International conference on military technologies (ICMT). IEEE, pp 1–5
DeBill E (2019) Module counts. http://www.modulecounts.com/
Decan A, Mens T, Constantinou E (2018) On the impact of security vulnerabilities in the npm package dependency network. In: Proceedings of the 15th international conference on mining software repositories, pp 181–191
Dey T, Mousavi S, Ponce E, Fry T, Vasilescu B, Filippova A, Mockus A (2020) Detecting and characterizing bots that commit code. In: Proceedings of the 17th international conference on mining software repositories, pp 209–219
Duan R, Alrawi O, Kasturi RP, Elder R, Saltaformaggio B, Lee W (2020) Towards measuring supply chain attacks on package managers for interpreted languages. arXiv:2002.01139
Erlenhov L, de Oliveira Neto FG, Scandariato R, Leitner P (2019b) Current and future bots in software development. In: 2019 IEEE/ACM 1st International workshop on bots in software engineering (BotSE). IEEE, pp 7–11
Erlenhov L, Gomes de Oliveira Neto F, Scandariato R, Leitner P (2019a) Current and future bots in software development. In: 2019 IEEE/ACM 1st International workshop on bots in software engineering (BotSE), pp 7–11. https://doi.org/10.1109/BotSE.2019.00009
Garrett K, Ferreira G, Jia L, Sunshine J, Kästner C (2019) Detecting suspicious package updates. In: Proceedings of the 41st International conference on software engineering: new ideas and emerging results. IEEE Press, ICSE-NIER ’19, p 13–16. https://doi.org/10.1109/ICSE-NIER.2019.00012
Gousios G, Pinzger M, Deursen Av (2014a) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering. Association for computing machinery, New York, NY, USA, ICSE 2014, pp 345–355. https://doi.org/10.1145/2568225.2568260
Gousios G, Pinzger M, Deursen Av (2014b) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering, pp 345–355
Gousios G, Zaidman A (2014) A dataset for pull-based development research. In: Proceedings of the 11th working conference on mining software repositories. Association for computing machinery, New York, NY, USA, MSR 2014, pp 368–371. https://doi.org/10.1145/2597073.2597122
R.M. Groves F.J. Fowler Jr M.P. Couper J.M. Lepkowski E. Singer R. Tourangeau Survey methodology, 2011 John Wiley & Sons
F. Hou S. Jansen A systematic literature review on trust in the software ecosystem Empir Softw Eng 2023 28 1 8 10.1007/s10664-022-10238-y
Imtiaz N, Khanom A, Williams L (2022) Open or sneaky? fast or slow? light or heavy?: Investigating security releases of open source packages. IEEE Trans Softw Eng
Jeong G, Kim S, Zimmermann T, Yi K (2009) Improving code review by predicting reviewers and acceptance of patches. Research on software analysis for error-free computing center Tech-Memo (ROSAEC MEMO 2009-006), pp 1–18
Kaczorowski M (2020) Secure at every step: what is software supply chain security and why does it matter? https://github.blog/2020-09-02-secure-your-software-supply-chain-and-protect-against-supply-chain-threats-github-blog/
Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining github. In: Proceedings of the 11th working conference on mining software repositories, pp 92–101
Kononenko O, Rose T, Baysal O, Godfrey M, Theisen D, De Water B (2018a) Studying pull request merges: a case study of shopify’s active merchant. In: Proceedings of the 40th international conference on software engineering: software engineering in practice, pp 124–133
Kononenko O, Rose T, Baysal O, Godfrey M, Theisen D, de Water B (2018b) Studying pull request merges: a case study of shopify’s active merchant. In: 2018 IEEE/ACM 40th International conference on software engineering: software engineering in practice track (ICSE-SEIP), pp 124–133
Ladisa P, Plate H, Martinez M, Barais O (2022) Taxonomy of attacks on open-source software supply chains. arXiv preprint arXiv:2204.04008
Lawall J, Muller G (2018) Coccinelle: 10 years of automated evolution in the Linux kernel. In: Proceedings of the 2018 USENIX conference on usenix annual technical conference. USENIX Association, USA, USENIX ATC ’18, pp 601–613
Lin G, Xiao W, Zhang J, Xiang Y (2019) Deep learning-based vulnerable function detection: a benchmark. In: International conference on information and communications security. Springer, pp 219–232
Lin G, Zhang J, Luo W, Pan L, Xiang Y (2017a) Poster: vulnerability discovery with function representation learning from unlabeled projects. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. Association for computing machinery, New York, NY, USA, CCS ’17, pp 2539–2541. https://doi.org/10.1145/3133956.3138840
Lin G, Zhang J, Luo W, Pan L, Xiang Y (2017b) Poster: vulnerability discovery with function representation learning from unlabeled projects. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp 2539–2541
Mirhosseini S, Parnin C (2017) Can automated pull requests encourage software developers to upgrade out-of-date dependencies? In: 2017 32nd IEEE/ACM international conference on automated software engineering (ASE). IEEE, pp 84–94
Moguel-Sánchez R, Martínez-Palacios CS, Ocharán-Hernández JO, Limón X, Sánchez-García ÁJ (2022) Bots and their uses in software development: a systematic mapping study. In: 2022 10th International conference in software engineering research and innovation (CONISOFT). IEEE, pp 140–149
S. Mujahid R. Abdalkareem E. Shihab What are the characteristics of highly-selected packages? a case study on the npm ecosystem J Syst Softw 2023 198 10.1016/j.jss.2022.111588
N. Munaiah S. Kroh C. Cabrey M. Nagappan Curating github for engineered software projects Empir Softw Eng 2017 22 3219 3253 10.1007/s10664-017-9512-6
Ohm M, Kempf L, Boes F, Meier M (2020a) Supporting the detection of software supply chain attacks through unsupervised signature generation. arXiv preprint arXiv:2011.02235
Ohm M, Kempf L, Boes F, Meier M (2021) Supporting the detection of software supply chain attacks through unsupervised signature generation. arXiv:2011.02235
Ohm M, Plate H, Sykosch A, Meier M (2020b) Backstabber’s knife collection: a review of open source software supply chain attacks. In: International conference on detection of intrusions and malware, and vulnerability assessment. Springer, pp 23–43
Pashchenko I, Vu DL, Massacci F (2020) A qualitative study of dependency management and its security implications. In: Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp 1513–1531
Peterson K (2013) The github open source development process. http://kevinp.me/github-process-research/github-processresearch.pdf (visited on 05/11/2017)
Pham R, Singer L, Liskin O, Figueira Filho F, Schneider K (2013) Creating a shared understanding of testing culture on a social coding site. In: 2013 35th International conference on software engineering (ICSE). IEEE, pp 112–121
Plumb T (2022) GitHub’s Octoverse report finds 97% of apps use open source software. https://venturebeat.com/programming-development/github-releases-open-source-report-octoverse-2022-says-97-of-apps-use-oss/
G.A.A. Prana A. Sharma L.K. Shar D. Foo A.E. Santosa A. Sharma D. Lo Out of sight, out of mind? how vulnerable dependencies affect open-source projects Empir Softw Eng 2021 26 4 1 34 10.1007/s10664-021-09959-3
Preston-Werner T (2021) Semantic versioning 2.0.0. https://semver.org/
Rigby PC, Bird C (2013) Convergent contemporary software peer review practices. In: Proceedings of the 2013 9th joint meeting on foundations of software engineering. Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2013, pp 202–212. https://doi.org/10.1145/2491411.2491444
Russell R, Kim L, Hamilton L, Lazovich T, Harer J, Ozdemir O, Ellingwood P, McConley M (2018) Automated vulnerability detection in source code using deep representation learning. In: 2018 17th IEEE international conference on machine learning and applications (ICMLA). IEEE, pp 757–762
S. Santhanam T. Hecking A. Schreiber S. Wagner Bots in software engineering: a systematic mapping study PeerJ Computer Science 2022 8 10.7717/peerj-cs.866
Soares DM, de Lima Júnior ML, Murta L, Plastino A (2015a) Acceptance factors of pull requests in open-source projects. In: Proceedings of the 30th annual ACM symposium on applied computing. Association for Computing Machinery, New York, USA, SAC ’15, pp 1541–1546. https://doi.org/10.1145/2695664.2695856
Soares DM, de Lima Júnior ML, Murta L, Plastino A (2015b) Acceptance factors of pull requests in open-source projects. In: Proceedings of the 30th annual ACM symposium on applied computing, pp 1541–1546
Soto-Valero C, Durieux T, Baudry B (2021) A longitudinal analysis of bloated java dependencies. In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering. Association for Computing Machinery, New York, USA, ESEC/FSE 2021, pp 1021–1031. https://doi.org/10.1145/3468264.3468589
Szulik K (2018) Dependency management and your software health. https://blog.tidelift.com/dependency-management-and-your-software-health
H. Wang G. Ye Z. Tang S.H. Tan S. Huang D. Fang Y. Feng L. Bian Z. Wang Combining graph-based learning with automated data collection for code vulnerability detection IEEE Trans Inf Forensics Secur 2020 16 1943 1958 10.1109/TIFS.2020.3044773
Wang Y, Chen B, Huang K, Shi B, Xu C, Peng X, Wu Y, Liu Y (2020b) An empirical study of usages, updates and risks of third-party libraries in java projects. In: 2020 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 35–45
Weißgerber P, Neu D, Diehl S (2008) Small patches get in! In: Proceedings of the 2008 international working conference on mining software repositories. Association for Computing Machinery, New York, USA, MSR ’08, pp 67–76. https://doi.org/10.1145/1370750.1370767
M. Wessel B.M. De Souza I. Steinmacher I.S. Wiese I. Polato A.P. Chaves M.A. Gerosa The power of bots: characterizing and understanding bots in oss projects Proc ACM Hum-Comput Interaction 2018 2 CSCW 1 19 10.1145/3274451
M. Wessel I. Wiese I. Steinmacher M.A. Gerosa Don’t disturb me: challenges of interacting with software bots on open source software projects Proc ACM Hum-Comput Interaction 2021 5 CSCW2 1 21 10.1145/3476042
Wessel M, Gerosa MA, Shihab E (2022) Software bots in software engineering: benefits and challenges. In: Proceedings of the 19th International conference on mining software repositories, pp 724–725
Wessel M, Steinmacher I (2020a) The inconvenient side of software bots on pull requests. In: Proceedings of the IEEE/ACM 42nd international conference on software engineering workshops. Association for Computing Machinery, New York, USA, CSEW’20, pp 51–55. https://doi.org/10.1145/3387940.3391504
Wessel M, Steinmacher I (2020b) The inconvenient side of software bots on pull requests. In: Proceedings of the IEEE/ACM 42nd international conference on software engineering workshops, pp 51–55
Yu Y, Wang H, Filkov V, Devanbu P, Vasilescu B (2015) Wait for it: determinants of pull request evaluation latency on github. In: 2015 IEEE/ACM 12th Working conference on mining software repositories, pp 367–371. https://doi.org/10.1109/MSR.2015.42
Zahan N, Zimmermann T, Godefroid P, Murphy B, Maddila C, Williams L (2022) What are weak links in the npm supply chain? In: 2022 IEEE/ACM 44th International conference on software engineering: software engineering in practice (ICSE-SEIP). IEEE, pp 331–340
A. Zerouali T. Mens A. Decan C. De Roover On the impact of security vulnerabilities in the npm and rubygems dependency networks Empir Softw Eng 2022 27 5 1 45 10.1007/s10664-022-10154-1
Zerouali A, Mens T, Decan A, Roover CD (2021) On the impact of security vulnerabilities in the npm and rubygems dependency networks. arXiv:2106.06747
Zhou Y, Liu S, Siow J, Du X, Liu Y (2019) Devign: effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Adv Neural Inf Process Syst 32
Zhu J, Zhou M, Mockus A (2016) Effectiveness of code contribution: from patch-based to pull-request-based tools. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering. Association for Computing Machinery, New York, USA, FSE 2016, pp 871–882. https://doi.org/10.1145/2950290.2950364
