Code Representation Learning; Commit Message Generation; Dataset Construction; Code changes; Code representation; Code representation learning; Commit message generation; Critical factors; Dataset construction; Language description; Learning Based Models; Natural languages; Template-based; Software; Safety, Risk, Reliability and Quality; Control and Optimization
Abstract :
[en] Commit message generation (CMG) is a challenging task in automated software engineering that aims to generate natural language descriptions of code changes for commits. Previous methods all start from the modified code snippets, outputting commit messages through template-based, retrieval-based, or learning-based models. While these methods can summarize what is modified from the perspective of code, they struggle to provide reasons for the commit. The correlation between commits and issues that could be a critical factor for generating rational commit messages is still unexplored. In this work, we delve into the correlation between commits and issues from the perspective of dataset and methodology. We construct the first dataset anchored on combining correlated commits and issues. The dataset consists of an unlabeled commit-issue parallel part and a labeled part in which each example is provided with human-annotated rational information in the issue. Furthermore, we propose ExGroFi (Extraction, Grounding, Ene-tuning), a novel paradigm that can introduce the correlation between commits and issues into the training phase of models. To evaluate whether it is effective, we perform comprehensive experiments with various state-of-the-art CMG models. The results show that compared with the original models, the performance of ExGroFi-enhanced models is significantly improved.
Disciplines :
Computer science
Author, co-author :
Wang, Liran; Beihang University, Beijing, China
TANG, Xunzhu ✱; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
He, Yichen; Beihang University, Beijing, China
Ren, Changyu; Beihang University, Beijing, China
Shi, Shuhua; Beihang University, Beijing, China
Yan, Chaoran; Beihang University, Beijing, China
Li, Zhoujun; Beihang University, Beijing, China
✱ These authors have contributed equally to this work.
External co-authors :
yes
Language :
English
Title :
Delving into Commit-Issue Correlation to Enhance Commit Message Generation Models
Publication date :
15 September 2023
Event name :
2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)
Event place :
Echternach, Lux
Event date :
11-09-2023 => 15-09-2023
Main work title :
Proceedings - 2023 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023
Publisher :
Institute of Electrical and Electronics Engineers Inc.
National Natural Science Foundation of China Fund of the State Key Laboratory of Software Development Environment European Research Council (ERC)
Funding text :
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62276017, U1636211, 61672081), the Fund of the State Key Laboratory of Software Development Environment (Grant No. SKLSDE-2021ZX-18) and the NATURAL project which has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant No. 949014).
R. P. Buse and W. R. Weimer, "Automatically documenting program changes, " in Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '10. New York, NY, USA: Association for Computing Machinery, 2010, p. 33-42. [Online]. Available: Https: //doi. org/10. 1145/1858996. 1859005
L. F. Cortés-Coy, M. Linares-Vásquez, J. Aponte, and D. Poshyvanyk, "On automatically generating commit messages via summarization of source code changes, " in 2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation, 2014, pp. 275-284.
Mockus and Votta, "Identifying reasons for software changes using historic databases, " in Proceedings 2000 International Conference on Software Maintenance, 2000, pp. 120-130.
Y. Tian, Y. Zhang, K.-J. Stol, L. Jiang, and H. Liu, "What makes a good commit message?" in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE '22. New York, NY, USA: Association for Computing Machinery, 2022, p. 2389-2401. [Online]. Available: Https: //doi. org/10. 1145/3510003. 3510205
R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen, "Boa: A language and infrastructure for analyzing ultra-large-scale software repositories, " in 2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 422-431.
P. Loyola, E. Marrese-Taylor, and Y. Matsuo, "A neural architecture for generating natural language descriptions from source code changes, " in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 287-292. [Online]. Available: Https: //aclanthology. org/P17-2045
W. Maalej and H.-J. Happel, "Can development work describe itself?" in 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), 2010, pp. 191-200.
S. Jiang, A. Armaly, and C. McMillan, "Automatically generating commit messages from diffs using neural machine translation, " in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), 2017, pp. 135-146.
W. Tao, Y. Wang, E. Shi, L. Du, S. Han, H. Zhang, D. Zhang, and W. Zhang, "On the evaluation of commit message generation models: An experimental study, " in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2021, pp. 126-136.
R. P. Buse and W. R. Weimer, "Automatically documenting program changes, " in Proceedings of the IEEE/ACM international conference on Automated software engineering, 2010, pp. 33-42.
J. Shen, X. Sun, B. Li, H. Yang, and J. Hu, "On automatic summarization of what and why information in source code changes, " in 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC), vol. 1. IEEE, 2016, pp. 103-112.
Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, "Neural-machine-translation-based commit message generation: How far are we?" in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ser. ASE '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 373-384. [Online]. Available: Https: //doi. org/10. 1145/3238147. 3238190
T. Hoang, H. J. Kang, D. Lo, and J. Lawall, "Cc2vec: Distributed representations of code changes, " in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE '20. New York, NY, USA: Association for Computing Machinery, 2020, p. 518-529. [Online]. Available: Https: //doi. org/10. 1145/3377811. 3380361
S. Xu, Y. Yao, F. Xu, T. Gu, H. Tong, and J. Lu, "Commit message generation for source code changes, " in IJCAI, 2019.
Q. Liu, Z. Liu, H. Zhu, H. Fan, B. Du, and Y. Qian, "Generating commit messages from diffs using pointer-generator network, " in 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), 2019, pp. 299-309.
L. Y. Nie, C. Gao, Z. Zhong, W. Lam, Y. Liu, and Z. Xu, "CoreGen: Contextualized Code Representation Learning for Commit Message Generation, " Neurocomputing, vol. 459, pp. 97-107, Oct. 2021. [Online]. Available: Https: //www. sciencedirect. com/science/article/pii/ S092523122100792X
J. Dong, Y. Lou, Q. Zhu, Z. Sun, Z. Li, W. Zhang, and D. Hao, "Fira: Fine-grained graph-based code change representation for automated commit message generation, " in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE '22. New York, NY, USA: Association for Computing Machinery, 2022, p. 970-981. [Online]. Available: Https: //doi. org/10. 1145/3510003. 3510069
S. Liu, C. Gao, S. Chen, L. Y. Nie, and Y. Liu, "Atom: Commit message generation based on abstract syntax tree and hybrid ranking, " IEEE Transactions on Software Engineering, vol. 48, no. 5, pp. 1800-1817, 2022.
E. Shi, Y. Wang, W. Tao, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, "Race: Retrieval-augmented commit message generation, " in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 5520-5530.
H. Wang, X. Xia, D. Lo, Q. He, X. Wang, and J. Grundy, "Contextaware retrieval-based deep commit message generation, " ACM Trans. Softw. Eng. Methodol., vol. 30, no. 4, jul 2021. [Online]. Available: Https: //doi. org/10. 1145/3464689
J. Li, T. Tang, W. X. Zhao, and J.-R. Wen, "Pretrained language model for text generation: A survey, " in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Z.-H. Zhou, Ed. International Joint Conferences on Artificial Intelligence Organization, 8 2021, pp. 4492-4499, survey Track. [Online]. Available: Https: //doi. org/10. 24963/ijcai. 2021/612
X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, "Pre-trained models for natural language processing: A survey, " Science China Technological Sciences, vol. 63, no. 10, pp. 1872-1897, 2020.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, "CodeBERT: A pre-trained model for programming and natural languages, " in Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics, Nov. 2020, pp. 1536-1547. [Online]. Available: Https: //aclanthology. org/2020. findings-emnlp. 139
A. Elnaggar, W. Ding, L. Jones, T. Gibbs, T. Feher, C. Angerer, S. Severini, F. Matthes, and B. Rost, "Codetrans: Towards cracking the language of silicone's code through self-supervised deep learning and high performance computing, " CoRR, vol. Abs/2104. 02443, 2021. [Online]. Available: Https: //arxiv. org/abs/2104. 02443
Y. Wang, W. Wang, S. Joty, and S. C. Hoi, "CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, " in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 8696-8708. [Online]. Available: Https: //aclanthology. org/2021. emnlp-main. 685
J. Anvik, L. Hiew, and G. C. Murphy, "Who should fix this bug?" in Proceedings of the 28th International Conference on Software Engineering, ser. ICSE '06. New York, NY, USA: Association for Computing Machinery, 2006, p. 361-370. [Online]. Available: Https: //doi. org/10. 1145/1134285. 1134336
J. Zhu, M. Zhou, and A. Mockus, "Effectiveness of code contribution: From patch-based to pull-request-based tools, " in Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE 2016. New York, NY, USA: Association for Computing Machinery, 2016, p. 871-882. [Online]. Available: Https: //doi. org/10. 1145/2950290. 2950364
N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann, "What makes a good bug report?" in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, 2008, pp. 308-318.
H. Tian, X. Tang, A. Habib, S. Wang, K. Liu, X. Xia, J. Klein, and T. F. Bissyand E, "Is this change the answer to that problem? correlating descriptions of bug and code changes for evaluating patch correctness, " in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE '22. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: Https: //doi. org/10. 1145/3551349. 3556914
R. Vieira, A. da Silva, L. Rocha, and J. A. P. Gomes, "From reports to bug-fix commits: A 10 years dataset of bug-fixing activity from 55 apache's open source projects, " in Proceedings of the Fifteenth International Conference on Predictive Models and Data Analytics in Software Engineering, ser. PROMISE'19. New York, NY, USA: ACM, 2019, pp. 80-89. [Online]. Available: Http: //doi. Acm. org/10. 1145/3345629. 3345639
Y. Zhou and A. Sharma, "Automated identification of security issues from commit messages and bug reports, " in Proceedings of the 2017 11th joint meeting on foundations of software engineering, 2017, pp. 914-919.
X. Tan and M. Zhou, "How to communicate when submitting patches: An empirical study of the linux kernel, " Proceedings of the ACM on Human-Computer Interaction, vol. 3, no. CSCW, pp. 1-26, 2019.
A. A. Sawant, G. Huang, G. Vilen, S. Stojkovski, and A. Bacchelli, "Why are features deprecated? An investigation into the motivation behind deprecation, " in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2018, pp. 13-24.
O. Baysal, R. Holmes, and M. W. Godfrey, "Situational awareness: Personalizing issue tracking systems, " in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 1185-1188.
J. Aranda and G. Venolia, "The secret life of bugs: Going past the errors and omissions in software repositories, " in 2009 IEEE 31st international conference on software engineering. IEEE, 2009, pp. 298-308.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding, " in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171-4186. [Online]. Available: Https: //aclanthology. org/N19-1423
A. Zagalsky, J. Feliciano, M.-A. Storey, Y. Zhao, and W. Wang, "The emergence of github as a collaborative platform for education, " in Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, 2015, pp. 1906-1917.
S. Onoue, H. Hata, and K.-i. Matsumoto, "A study of the characteristics of developers' activities in github, " in 2013 20th Asia-Pacific Software Engineering Conference (APSEC), vol. 2, 2013, pp. 7-12.
Y. Li, M. Soliman, and P. Avgeriou, "Identifying self-admitted technical debt in issue tracking systems using machine learning, " Empirical Software Engineering, vol. 27, no. 6, p. 131, 2022.
S. Breu, R. Premraj, J. Sillito, and T. Zimmermann, "Frequently asked questions in bug reports, " University of Calgary, Tech. Rep., 2009.
G. Destefanis, M. Ortu, D. Bowes, M. Marchesi, and R. Tonelli, "On measuring affects of github issues' commenters, " in Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering, ser. SEmotion '18. New York, NY, USA: Association for Computing Machinery, 2018, p. 14-19. [Online]. Available: Https: //doi. org/10. 1145/3194932. 3194936
PyGithub Contributors, "Pygithub: A python library to access the github api v3, " 2021, accessed: 2023-04-22. [Online]. Available: Https: //pygithub. readthedocs. io/en/latest/index. html
T.-D. B. Le, M. Linares-Vásquez, D. Lo, and D. Poshyvanyk, "Rclinker: Automated linking of issue reports and commits leveraging rich contextual information, " in 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 2015, pp. 36-47.
Z. Liao, D. He, Z. Chen, X. Fan, Y. Zhang, and S. Liu, "Exploring the characteristics of issue-related behaviors in github using visualization techniques, " IEEE Access, vol. 6, pp. 24 003-24 015, 2018.
I. Heartex, "Label studio: Open source data labeling and annotation tool, " 2021, accessed: 2023-04-22. [Online]. Available: Https: //labelstud. io
A. Akbik, D. Blythe, and R. Vollgraf, "Contextual string embeddings for sequence labeling, " in Proceedings of the 27th international conference on computational linguistics, 2018, pp. 1638-1649.
F. Murtagh, "Multilayer perceptrons for classification and regression, " Neurocomputing, vol. 2, no. 5-6, pp. 183-197, 1991.
G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, "Neural architectures for named entity recognition, " in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, California: Association for Computational Linguistics, Jun. 2016, pp. 260-270. [Online]. Available: Https: //aclanthology. org/N16-1030
N. Alshammari and S. Alanazi, "The impact of using different annotation schemes on named entity recognition, " Egyptian Informatics Journal, vol. 22, no. 3, pp. 295-302, 2021.
K. R. Chandu, Y. Bisk, and A. W. Black, "Grounding 'grounding' in NLP, " in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Online: Association for Computational Linguistics, Aug. 2021, pp. 4283-4305. [Online]. Available: Https: //aclanthology. org/2021. findings-acl. 375
I. Sutskever, O. Vinyals, and Q. V. Le, "Sequence to sequence learning with neural networks, " Advances in neural information processing systems, vol. 27, 2014.
D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, "UniXcoder: Unified cross-modal pre-training for code representation, " in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 7212-7225. [Online]. Available: Https: //aclanthology. org/2022. Acl-long. 499
Hugging Face, "Hugging face: Natural language processing and machine learning platform, " 2021, accessed: 2023-04-22. [Online]. Available: Https: //huggingface. co
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need, " Advances in neural information processing systems, vol. 30, 2017.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: A method for automatic evaluation of machine translation, " in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311-318.
C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries, " in Text summarization branches out, 2004, pp. 74-81.
M. Paterson and V. Dancík, "Longest common subsequences, " in Mathematical Foundations of Computer Science 1994: 19th International Symposium, MFCS'94 Kosice, Slovakia, August 22-26, 1994 Proceedings 19. Springer, 1994, pp. 127-142.
S. Banerjee and A. Lavie, "Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, " in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65-72.
R. Vedantam, C. Lawrence Zitnick, and D. Parikh, "Cider: Consensusbased image description evaluation, " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 4566-4575.
Z. Nasar, S. W. Jaffry, and M. K. Malik, "Named entity recognition and relation extraction: State-of-the-art, " ACM Comput. Surv., vol. 54, no. 1, feb 2021. [Online]. Available: Https: //doi. org/10. 1145/3445965
P. E. McKnight and J. Najab, "Mann-whitney u test, " The Corsini encyclopedia of psychology, pp. 1-1, 2010.
J. L. Fleiss and J. Cohen, "The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability, " Educational and psychological measurement, vol. 33, no. 3, pp. 613-619, 1973.
N. Dragan, M. L. Collard, M. Hammad, and J. I. Maletic, "Using stereotypes to help characterize commits, " in 2011 27th IEEE International Conference on Software Maintenance (ICSM). IEEE, 2011, pp. 520-523.
N. Dragan, M. L. Collard, and J. I. Maletic, "Reverse engineering method stereotypes, " in 2006 22nd IEEE International Conference on Software Maintenance. IEEE, 2006, pp. 24-34.
T. Luong, H. Pham, and C. D. Manning, "Effective approaches to attention-based neural machine translation, " in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 1412-1421. [Online]. Available: Https: //aclanthology. org/D15-1166
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip, "A comprehensive survey on graph neural networks, " IEEE transactions on neural networks and learning systems, vol. 32, no. 1, pp. 4-24, 2020.
