Data augmentation; Manifold-mixup; Graph neural networks
Résumé :
[en] Graph neural network (GNN)-based graph learning has been popular in natural language and programming language processing, particularly in text and source code classification. Typically, GNNs are constructed by incorporating alternating layers which learn transformations of graph node features, along with graph pooling layers that use graph pooling operators (e.g., Max-pooling) to effectively reduce the number of nodes while preserving the semantic information of the graph. Recently, to enhance GNNs in graph learning tasks, Manifold-Mixup, a data augmentation technique that produces synthetic graph data by linearly mixing a pair of graph data and their labels, has been widely adopted. However, the performance of Manifold-Mixup can be highly affected by graph pooling operators, and there have not been many studies that are dedicated to uncovering such affection. To bridge this gap, we take an early step to explore how graph pooling operators affect the performance of Mixup-based graph learning. To that end, we conduct a comprehensive empirical study by applying Manifold-Mixup to a formal characterization of graph pooling based on 11 graph pooling operations (9 hybrid pooling operators, 2 non-hybrid pooling operators). The experimental results on both natural language datasets (Gossipcop, Politifact) and programming language datasets (JAVA250, Python800) demonstrate that hybrid pooling operators are more effective for Manifold-Mixup than the standard Max-pooling and the state-of-the-art graph multiset transformer (GMT) pooling, in terms of producing more accurate and robust GNN models. Editor's note: Open Science material was validated by the Journal of Systems and Software Open Science Board.
Disciplines :
Sciences informatiques
Auteur, co-auteur :
DONG, Zeming ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal ; Kyushu University, Japan
HU, Qiang ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust > SerVal > Team Yves LE TRAON
Zhang, Zhenya ; Kyushu University, Japan
GUO, Yuejun ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust > SerVal > Team Yves LE TRAON ; Luxembourg Institute of Science and Technology, Luxembourg
CORDY, Maxime ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
PAPADAKIS, Mike ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal
Traon, Yves Le; University of Luxembourg, Luxembourg
Zhao, Jianjun ; Kyushu University, Japan
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
On the effectiveness of hybrid pooling in mixup-based graph learning for language processing
JST-Mirai Program Japan Society for the Promotion of Science
Subventionnement (détails) :
This research is supported in part by JSPS KAKENHI Grant No. JP23H03372 , Japan.This research is supported in part by JSPS KAKENHI Grant No. JP23H03372 and No. JP24K02920, Japan. The research is also supported in part by JST-Mirai Program Grant No. JPMJMI20B8.
Allamanis, M., Barr, E.T., Devanbu, P., Sutton, C., A survey of machine learning for big code and naturalness. ACM Comput. Surv., 51(4), 2018, 81.
Allamanis, M., Brockschmidt, M., Khademi, M., Learning to represent programs with graphs. International Conference on Learning Representations, 2018 URL https://openreview.net/forum?id=BJOFETxR-.
Allamanis, M., Jackson-Flux, H., Brockschmidt, M., Self-supervised bug detection and repair. Adv. Neural Inf. Process. Syst. 34 (2021), 27865–27876.
Armstrong, R.A., When to use the B onferroni correction. Ophthalmic Physiol. Opt. 34:5 (2014), 502–508.
Atwood, J., Towsley, D., Diffusion-convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS ’16, 2016, Curran Associates Inc., Red Hook, NY, USA, 2001–2009.
Baek, J., Kang, M., Hwang, S.J., Accurate learning of graph representations with graph multiset pooling. International Conference on Learning Representations, 2021 URL https://openreview.net/forum?id=JHcqXGaqiGn.
Bahdanau, D., Cho, K., Bengio, Y., Neural machine translation by jointly learning to align and translate. 2014 arXiv preprint arXiv:1409.0473.
Bianchi, F.M., Grattarola, D., Alippi, C., Spectral clustering with graph neural networks for graph pooling. International Conference on Machine Learning, 2020, PMLR, 874–883.
Chen, J., Tam, D., Raffel, C., Bansal, M., Yang, D., An empirical survey of data augmentation for limited data learning in NLP. Trans. Assoc. Comput. Linguist. 11 (2023), 191–211.
Chen, J., Yang, Z., Yang, D., MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, Association for Computational Linguistics, Online, 2147–2157, 10.18653/v1/2020.acl-main.194 URL https://aclanthology.org/2020.acl-main.194.
Dinella, E., Dai, H., Li, Z., Naik, M., Song, L., Wang, K., 2020. Hoppity: Learning graph transformations to detect and fix bugs in programs. In: International Conference on Learning Representations. ICLR.
Dong, Z., Hu, Q., Guo, Y., Cordy, M., Papadakis, M., Zhang, Z., Traon, Y.L., Zhao, J., MixCode: Enhancing code classification by mixup-based data augmentation. 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER, 2023, 379–390, 10.1109/SANER56733.2023.00043.
Dong, Z., Hu, Q., Guo, Y., Zhang, Z., Cordy, M., Papadakis, M., Traon, Y.L., Zhao, J., Boosting source code learning with data augmentation: An empirical study. 2023 arXiv preprint arXiv:2303.06808.
Dong, Z., Hu, Q., Zhang, Z., Zhao, J., On the effectiveness of graph data augmentation for source code learning. Knowl.-Based Syst., 285, 2024, 111328, 10.1016/j.knosys.2023.111328.
Dou, Y., Shu, K., Xia, C., Yu, P.S., Sun, L., User preference-aware fake news detection. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, 2021, Association for Computing Machinery, New York, NY, USA, 2051–2055, 10.1145/3404835.3462990.
Fabian, B., Baumann, A., Lackner, J., Topological analysis of cloud service connectivity. Comput. Ind. Eng. 88:C (2015), 151–165, 10.1016/j.cie.2015.06.009.
Feng, W., Zhang, J., Dong, Y., Han, Y., Luan, H., Xu, Q., Yang, Q., Kharlamov, E., Tang, J., Graph random neural networks for semi-supervised learning on graphs. NeurIPS, vol. 33, 2020, 22092–22103.
Fey, M., Lenssen, J.E., 2019. Fast Graph Representation Learning with PyTorch Geometric. In: ICLR Workshop on Representation Learning on Graphs and Manifolds.
Gao, H., Ji, S., Graph U-Nets. International Conference on Machine Learning, 2019, PMLR, 2083–2092.
Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E., Neural message passing for quantum chemistry. Precup, D., Teh, Y.W., (eds.) Proceedings of the 34th International Conference on Machine Learning Proceedings of Machine Learning Research, vol. 70, 2017, PMLR, 1263–1272 URL https://proceedings.mlr.press/v70/gilmer17a.html.
Grattarola, D., Zambon, D., Bianchi, F.M., Alippi, C., Understanding pooling in graph neural networks. IEEE Trans. Neural Netw. Learn. Syst., 2022, 1–11, 10.1109/TNNLS.2022.3190922.
Guo, H., Mao, Y., Zhang, R., Augmenting data with mixup for sentence classification: An empirical study. 2019 arXiv preprint arXiv:1905.08941.
Hamilton, W., Ying, Z., Leskovec, J., Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, vol. 30, 2017.
Huang, L., Ma, D., Li, S., Zhang, X., Wang, H., Text level graph neural network for text classification. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, 2019, Association for Computational Linguistics, Hong Kong, China, 3444–3450, 10.18653/v1/D19-1345 URL https://aclanthology.org/D19-1345.
Kingma, D.P., Ba, J., Adam: A method for stochastic optimization. 2014 arXiv preprint arXiv:1412.6980.
Kipf, T.N., Welling, M., Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017 URL https://openreview.net/forum?id=SJU4ayYgl.
Knyazev, B., Taylor, G.W., Amer, M.R., Understanding attention and generalization in graph neural networks. Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, Curran Associates Inc., Red Hook, NY, USA.
Konno, R., Matsubayashi, Y., Kiyono, S., Ouchi, H., Takahashi, R., Inui, K., An empirical study of contextual data augmentation for Japanese zero anaphora resolution. Proceedings of the 28th International Conference on Computational Linguistics, 2020, International Committee on Computational Linguistics, Barcelona, Spain (Online), 4956–4968, 10.18653/v1/2020.coling-main.435 URL https://aclanthology.org/2020.coling-main.435.
Lee, J., Lee, I., Kang, J., Self-attention graph pooling. International Conference on Machine Learning, 2019, PMLR, 3734–3743.
Li, J., Rong, Y., Cheng, H., Meng, H., Huang, W., Huang, J., 2019. Semi-supervised graph classification: A hierarchical graph perspective. In: The World Wide Web Conference. pp. 972–982.
Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.S., Gated graph sequence neural networks. Bengio, Y., LeCun, Y., (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016 URL http://arxiv.org/abs/1511.05493.
Ma, Y., Wang, S., Aggarwal, C.C., Tang, J., 2019. Graph convolutional networks with eigenpooling. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 723–731.
Mesquita, D., Souza, A.H., Kaski, S., Rethinking pooling in graph neural networks. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, 2020, Curran Associates Inc., Red Hook, NY, USA.
Neyshabur, B., Bhojanapalli, S., McAllester, D., Srebro, N., Exploring generalization in deep learning. Advances in Neural Information Processing Systems, vol. 30, 2017.
Nguyen, V.-A., Nguyen, V., Le, T., Tran, Q.H., Phung, D., et al. ReGVD: Revisiting graph neural networks for vulnerability detection. 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings, ICSE-Companion, 2022, IEEE, 178–182.
Oehlers, M., Fabian, B., Graph metrics for network robustness—A survey. Mathematics, 9(8), 2021, 10.3390/math9080895 URL https://www.mdpi.com/2227-7390/9/8/895.
Papp, P.A., Martinkus, K., Faber, L., Wattenhofer, R., DropGNN: Random dropouts increase the expressiveness of graph neural networks. Adv. Neural Inf. Process. Syst. 34 (2021), 21997–22009.
Puri, R., Kung, D.S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. 2021 arXiv preprint arXiv:2105.12655.
Ranjan, E., Sanyal, S., Talukdar, P., Asap: Adaptive structure aware pooling for learning hierarchical graph representations. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2020, 5470–5477.
Rong, Y., Huang, W., Xu, T., Huang, J., 2020. DropEdge: towards deep graph convolutional networks on node classification. In: ICLR.
Shorten, C., Khoshgoftaar, T.M., A survey on image data augmentation for deep learning. J. Big Data 6:1 (2019), 1–48.
Simonovsky, M., Komodakis, N., 2017. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3693–3702.
Sun, L., Xia, C., Yin, W., Liang, T., Yu, P., He, L., Mixup-transformer: Dynamic data augmentation for NLP tasks. Proceedings of the 28th International Conference on Computational Linguistics, 2020, International Committee on Computational Linguistics, Barcelona, Spain (Online), 3436–3440, 10.18653/v1/2020.coling-main.305 URL https://aclanthology.org/2020.coling-main.305.
Tu, Y., How robust is the internet?. Nature 406:6794 (2000), 353–354.
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y., Graph attention networks. stat, 1050, 2017, 20.
Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., Bengio, Y., Manifold mixup: Better representations by interpolating hidden states. Chaudhuri, K., Salakhutdinov, R., (eds.) Proceedings of the 36th International Conference on Machine Learning Proceedings of Machine Learning Research, vol. 97, 2019, PMLR, 6438–6447 URL https://proceedings.mlr.press/v97/verma19a.html.
Wang, Y.G., Li, M., Ma, Z., Montúfar, G., Zhuang, X., Fan, Y., Haarpooling: Graph pooling with compressive haar basis. 2019.
Wang, W., Li, G., Ma, B., Xia, X., Jin, Z., Detecting code clones with graph neural network and flow-augmented abstract syntax tree. 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering, SANER, 2020, 261–271, 10.1109/SANER48275.2020.9054857.
Wang, Y., Wang, W., Liang, Y., Cai, Y., Hooi, B., 2021. Mixup for node and graph classification. In: Proceedings of the Web Conference 2021. pp. 3663–3674.
Xu, K., Hu, W., Leskovec, J., Jegelka, S., How powerful are graph neural networks?. International Conference on Learning Representations, 2019 URL https://openreview.net/forum?id=ryGs6iA5Km.
Yao, L., Mao, C., Luo, Y., Graph convolutional networks for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, 7370–7377.
Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., Leskovec, J., Hierarchical graph representation learning with differentiable pooling. Advances in Neural Information Processing Systems, vol. 31, 2018.
Yoon, S., Kim, G., Park, K., SSMix: Saliency-based span mixup for text classification. Findings of the Association for Computational Linguistics, ACL-IJCNLP 2021, 2021, Association for Computational Linguistics, Online, 3225–3234, 10.18653/v1/2021.findings-acl.285 URL https://aclanthology.org/2021.findings-acl.285.
Yu, S., Wang, T., Wang, J., Data augmentation by program transformation. J. Syst. Softw., 190, 2022, 111304, 10.1016/j.jss.2022.111304 URL https://www.sciencedirect.com/science/article/pii/S0164121222000541.
Yuan, H., Ji, S., 2020. Structpool: Structured graph pooling via conditional random fields. In: Proceedings of the 8th International Conference on Learning Representations.
Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y., 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6023–6032.
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D., mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018 URL https://openreview.net/forum?id=r1Ddp1-Rb.
Zhang, M., Cui, Z., Neumann, M., Chen, Y., An end-to-end deep learning architecture for graph classification. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018.
Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., Zou, J., How does mixup help with robustness and generalization?. International Conference on Learning Representations, 2021 URL https://openreview.net/forum?id=8yKEo06dKNo.
Zhang, L., Deng, Z., Kawaguchi, K., Zou, J., When and how mixup improves calibration. Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S., (eds.) Proceedings of the 39th International Conference on Machine Learning Proceedings of Machine Learning Research, vol. 162, 2022, PMLR, 26135–26160 URL https://proceedings.mlr.press/v162/zhang22f.html.
Zhang, Y., Yu, X., Cui, Z., Wu, S., Wen, Z., Wang, L., Every document owns its structure: Inductive text classification via graph neural networks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, Association for Computational Linguistics, Online, 334–339, 10.18653/v1/2020.acl-main.31 URL https://aclanthology.org/2020.acl-main.31.
Zhang, R., Yu, Y., Zhang, C., SeqMix: Augmenting active sequence labeling via sequence mixup. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2020, Association for Computational Linguistics, Online, 8566–8579, 10.18653/v1/2020.emnlp-main.691 URL https://aclanthology.org/2020.emnlp-main.691.
Zhao, T., Jin, W., Liu, Y., Wang, Y., Liu, G., Günneman, S., Shah, N., Jiang, M., Graph data augmentation for graph machine learning: A survey. IEEE Data Eng. Bull., 2023.
Zhou, Y., Liu, S., Siow, J., Du, X., Liu, Y., Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, Curran Associates Inc, Red Hook, NY, USA.
