[en] Code comment generation, which attempts to summarize the functionality of source code in textual descriptions, plays an important role in automatic software development research. Currently, several structural neural networks have been exploited to preserve the syntax structure of source code based on abstract syntax trees (ASTs). However, they can not well capture both the long-distance and local relations between nodes while retaining the overall structural information of AST. To mitigate this problem, we present a prototype tool titled PyScribe, which extends the Transformer model to a new encoder-decoder-based framework. Particularly, the triplet position is designed and integrated into the node-level and edge-level structural features of AST for producing Python code comments automatically. This paper, to the best of our knowledge, makes the first effort to model the edges of AST as an explicit component for improved code representation. By specifying triplet positions for each node and edge, the overall structural information can be well preserved in the learning process. Moreover, the captured node and edge features go through a two-stage decoding process to yield higher qualified comments. To evaluate the effectiveness of PyScribe, we resort to a large dataset of code-comment pairs by mining Jupyter Notebooks from GitHub, for which we have made it publicly available to support further studies. The experimental results reveal that PyScribe is indeed effective, outperforming the state-ofthe-art by achieving an average BLEU score (i.e., av-BLEU) of (Formula presented.) 0.28.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Other
Disciplines :
Computer science
Author, co-author :
Guo, Juncai ; School of Computer Science, Wuhan University, Wuhan, China
Liu, Jin; School of Computer Science, Wuhan University, Wuhan, China ; Key Laboratory of Network Assessment Technology, Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Liu, Xiao; School of Information Technology, Deakin University, Burwood, Melbourne, Australia
Wan, Yao; School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
Zhao, Yanjie; Faculty of Information Technology, Monash University, Clayton, Australia
Li, Li; School of Software, Beihang University, Beijing, China
Liu, Kui; Huawei Software Engineering Application Technology LabHangzhou, China
KLEIN, Jacques ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
China Scholarship Council National Natural Science Foundation of China
Funding text :
This research is supported by the National Natural Science Foundation of China (Grant No.61972290), the Open Fund of Key Laboratory of Network Assessment Technology from Chinese Academy of Sciences, and the China Scholarship Council (Grant No.201906270158). Open access publishing facilitated by Deakin University, as part of the Wiley ‐ Deakin University agreement via the Council of Australian University Librarians.
Stueben M. Self-DOCUMENTING CODE: 67–90. Apress; 2018.
Raskin J. Comments Are More Important Than Code. ACM Queue. 2005;3:64. doi:10.1145/1053331.1053354
McBurney PW, McMillan C. Automatic documentation generation via source code summarization of method context. In: Roy CK, Begel A, Moonen L, eds. 22nd International Conference on Program Comprehension, ICPC 2014, Hyderabad, India, June 2-3, 2014. ACM; 2014:279-290.
Sridhara G, Hill E, Muppaneni D, Pollock L, Vijay-Shanker K. Towards automatically generating summary comments for Java methods. Proceedings of the IEEE/ACM international conference on Automated software engineering. ACM; 2010:43-52.
Sridhara G, Pollock L, Vijay-Shanker K. Automatically detecting and describing high level actions within methods. 2011 33rd International Conference on Software Engineering (ICSE). ACM; 2011:101-110.
Robillard P. Schematic pseudocode for program constructs and its computer automation by SCHEMACODE. Commun ACM. 1986;29:1072-1089. doi:10.1145/7538.7541
Ohba M, Gondow K. Toward mining “concept keywords” from identifiers in large software projects. ACM SIGSOFT Softw Eng Notes. 2005;30(4):1-5. doi:10.1145/1082983.1083151
Maskeri G, Sarkar S, Heafield K. Mining business topics in source code using latent dirichlet allocation. Proceedings of the 2008 1st India Software Engineering Conference, ISEC'08. ACM; 2008:113-120.
Haiduc S, Aponte J, Marcus A. Supporting program comprehension with source code summarization. Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering. ACM; 2010:223-226.
Wong E, Yang J, Tan L. AutoComment: Mining question and answer sites for automatic comment generation. 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE; 2013:562-567.
Wong E, Liu T, Tan L. CloCom: Mining existing source code for automatic comment generation. 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE Computer Society; 2015:380-389.
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems. Curran Associates, Inc; 2017:5998-6008 http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf
Hu X, Li G, Xia X, Lo D, Lu S, Jin Z. Summarizing source code with transferred API knowledge. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018. ijcai.org; 2018:2269-2275.
Wei B, Li G, Xia X, Fu Z, Jin Z. Code generation as a dual task of code summarization. Advances in Neural Information Processing Systems. Curran Associates, Inc; 2019:6563-6573 http://arxiv.org/abs/1910.05923
Ahmad W, Chakraborty S, Ray B, Chang KW. A Transformer-based approach for source code summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2020:4998-5007.
Hu X, Li G, Xia X, Lo D, Jin Z. Deep code comment generation. ICPC '18: Proceedings of the 26th Conference on Program Comprehension. ACM; 2018:200-210.
Hu X, Li G, Xia X, Lo D, Jin Z. Deep code comment generation with hybrid lexical and syntactical information. Empir Softw Eng. 2020;25:2179-2217. doi:10.1007/s10664-019-09730-9
Alon U, Brody S, Levy O, Yahav E. code2seq: Generating sequences from structured representations of code. 7th International Conference on Learning Representations (ICLR). OpenReview.net; 2019.
LeClair A, Haque S, Wu L, McMillan C. Improved code summarization via a graph neural network. 2020 IEEE/ACM International Conference on Program Comprehension (ICPC). ACM; 2020:184-195.
Liu S, Chen Y, Xie X, Siow J, Liu Y. Automatic code summarization via multi-dimensional semantic fusing in GNN. CoRR. 2020; abs/2006.05405; https://arxiv.org/abs/2006.05405
Wan Y, Zhao Z, Yang M, et al. Improving automatic source code summarization via deep reinforcement learning. Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM; 2018:397-407.
Liang Y, Zhu KQ. Automatic generation of text descriptive comments for code blocks. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18). AAAI Press; 2018:5229-5236.
Gehring J, Auli M, Grangier D, Yarats D, Dauphin YN. Convolutional sequence to sequence learning. Proceedings of the 34th International Conference on Machine Learning. Vol 70. PMLR; 2017:1243-1252.
Barone AVM, Sennrich R. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017. Asian Federation of Natural Language Processing; 2017:314-319.
LeClair A, Jiang S, McMillan C. A neural model for generating natural language summaries of program subroutines. 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE / ACM; 2019:795-806.
Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code. CoRR. 2021; abs/2107.03374.
Wang Y, Wang W, Joty SR, Hoi SCH. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021. Association for Computational Linguistics; 2021:8696-8708.
OpenAI. ChatGPT. Computer Software; 2023.
Iyer S, Konstas I, Cheung A, Zettlemoyer L. Summarizing source code using a neural attention model. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The Association for Computer Linguistics; 2016:2073-2083.
Tian H, Liu K, Kaboré AK, et al. Evaluating representation learning of code changes for predicting patch correctness in program repair. 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. IEEE; 2020:981-992.
Wan Y, Shu J, Sui Y, et al. Multi-modal attention network learning for semantic source code retrieval. 34th IEEE/ACM International Conference on Automated Software Engineering, ASE 2019. IEEE; 2019:13-25.
Shido Y, Kobayashi Y, Yamamoto A, Miyamoto A, Matsumura T. Automatic source code summarization with extended tree-LSTM. 2019 International Joint Conference on Neural Networks (IJCNN). IEEE; 2019:1-8.
Zhou Z, Yu H, Fan G. Effective approaches to combining lexical and syntactical information for code summarization. Softw Pract Exp. 2020;50(12):2313-2336. doi:10.1002/spe.2893
Choi Y, Bak J, Na C, Lee J. Learning sequential and structural information for source code summarization. Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021. Association for Computational Linguistics; 2021:2842-2851.
Wang X, Wang Y, Mi F, et al. SynCoBERT: Syntax-guided multi-modal contrastive pre-training for code representation. arXiv preprint, arXiv:2108.04556 2021.
Zhao Y, Li L, Wang H, He Q, Grundy J. APIMatchmaker: Matching the right APIs for supporting the development of android apps. IEEE Trans Softw Eng. 2022;49:113-130.
Zhao Y, Li L, Sun X, Liu P, Grundy JC. Icon2Code: Recommending code implementations for Android GUI components. Inf Softw Technol. 2021;138:106619. doi:10.1016/j.infsof.2021.106619
Allamanis M, Peng H, Sutton C. A convolutional attention network for extreme summarization of source code. In: Balcan M, Weinberger KQ, eds. Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016. Vol 48. JMLR.org; 2016:2091-2100.
Wei H, Li M. Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code. Proceedings of the 26th International Joint Conference on Artificial Intelligence. ijcai.org; 2017:3034-3040.
Cho K, Merrienboer vB, Gülçehre Ç, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. ACL; 2014:1724-1734.
Kipf TN, Welling M. Semi-supervised classification with graph convolutional networks. Proceedings of the 5th International Conference on Learning Representations. OpenReview.net; 2017.
Wang W, Zhang Y, Sui Y, et al. Reinforcement-learning-guided source code summarization via hierarchical attention. IEEE Trans Softw Eng. 2020;48:102-119. doi:10.1109/TSE.2020.2979701
Bird S. NLTK: The natural language toolkit. ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. The Association for Computer Linguistics; 2006:69-72.
Liu K, Kim D, Bissyandé TF, Yoo S, Le Traon Y. Mining fix patterns for FindBugs violations. IEEE Trans Softw Eng. 2018;47(1):165-188. doi:10.1109/TSE.2018.2884955
Liu K, Kim D, Bissyandé TF, et al. Learning to spot and refactor inconsistent method names. Proceedings of the 41st International Conference on Software Engineering. IEEE; 2019:1-12.
Mou L, Li G, Zhang L, Wang T, Jin Z. Convolutional neural networks over tree structures for programming language processing. Proceedings of the 30th AAAI Conference on Artificial Intelligence. AAAI Press; 2016:1287-1293.
Zhang J, Wang X, Zhang H, Sun H, Wang K, Liu X. A novel neural source code representation based on abstract syntax tree. Proceedings of the IEEE/ACM 41st International Conference on Software Engineering. IEEE / ACM; 2019:783-794.
Le QV, Mikolov T. Distributed representations of sentences and documents. Proceedings of the 31st International Conference on Machine Learning. JMLR.org; 2014:1188-1196.
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society; 2016:770-778.
Sun Z, Li L, Liu Y, Du X, Li L. On the importance of building high-quality training datasets for neural code search. The 44th International Conference on Software Engineering (ICSE 2022). ACM: 2022.
Repplinger J, Chowdhury GG. Introduction to Modern Information Retrieval. 3rd ed.; 2019 Coll Res Libr, Facet publishing: 2011;72(2):194-195. http://crl.acrl.org/content/72/2/194.full.pdf
Kingma DP, Ba J. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations. OpenReview.net; arXiv: 2015.
Papineni K, Roukos S, Ward T, Zhu W. Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. ACL; 2002:311-318.
Banerjee S, Lavie A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics; 2005:65-72.
Lin CY. ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the ACL Workshop: Text Summarization Braches Out 2004. Association for Computational Linguistics; 2004:74-81.
See A, Liu PJ, Manning CD. Get to the point: summarization with pointer-generator networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2017:1073-1083.
Zhang J, Wang X, Zhang H, Sun H, Liu X. Retrieval-based neural source code summarization. ICSE '20: 42nd International Conference on Software Engineering. ACM; 2020:1385-1397.
Eriguchi A, Hashimoto K, Tsuruoka Y. Tree-to-sequence attentional neural machine translation. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. The Association for Computer Linguistics; 2016:823-833.
Wu H, Zhao H, Zhang M. Code Summarization with Structure-induced Transformer. ACL/IJCNLP 2021 of Findings of ACL. Association for Computational Linguistics; 2021:1078-1090.
Feng Z, Guo D, Tang D, et al. CodeBERT: A pre-trained model for programming and natural languages. Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics; 2020:1536-1547.
Liu Y, Ott M, Goyal N, et al. RoBERTa: A robustly optimized BERT pretraining approach. CoRR. 2019; abs/1907.11692.
Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res. 2020;21(140):1-67.
Wang Y, Dong Y, Lu X, Zhou A. GypSum: Learning hybrid representations for code summarization. In Rastogi A, Tufano R, Bavota G, Arnaoudova V, Haiduc S. eds. Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension. ICPC 2022, Virtual Event, May 16-17. ACM; 2022:abs/2204.12916. doi:10.48550/arXiv.2204.12916
Devlin J, Chang M, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019. Association for Computational Linguistics; 2019:4171-4186.
Zhao Y, Li L, Wang H, et al. On the impact of sample duplication in machine-learning-based android malware detection. ACM Trans Softw Eng Methodol. 2021;30(3):40:1-40:38. doi:10.1145/3446905
Tao W, Wang Y, Shi E, et al. On the evaluation of commit message generation models: An experimental study. IEEE International Conference on Software Maintenance and Evolution, ICSME 2021. IEEE; 2021:126-136.
LeClair A, McMillan C. Recommendations for datasets for source code summarization. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019. Association for Computational Linguistics; 2019:3931-3937.
Allamanis M. The adverse effects of code duplication in machine learning models of code. In: Masuhara H, Petricek T, eds. Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2019, Athens, Greece, October 23-24, 2019. ACM; 2019:143-153.
