Paper published in a book (Scientific congresses, symposiums and conference proceedings)
Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Space
PHILIPPY, Fred; Guo, Siwen; Haddadan, Shohreh
2023 • In Beinborn, Lisa; Goswami, Koustava; Murado{\u{g}}lu, Salihaet al. (Eds.) Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP
[en] Prior research has investigated the impact of various linguistic features on cross-lingual transfer performance. In this study, we investigate the manner in which this effect can be mapped onto the representation space. While past studies have focused on the impact on cross-lingual alignment in multilingual language models during fine-tuning, this study examines the absolute evolution of the respective language representation spaces produced by MLLMs. We place a specific emphasis on the role of linguistic characteristics and investigate their inter-correlation with the impact on representation spaces and cross-lingual transfer performance. Additionally, this paper provides preliminary evidence of how these findings can be leveraged to enhance transfer to linguistically distant languages.
Disciplines :
Computer science
Author, co-author :
PHILIPPY, Fred ; University of Luxembourg ; Zortify S.A. > Zortify Labs
Guo, Siwen; Zortify S.A. > Zortify Labs
Haddadan, Shohreh; Zortify S.A. > Zortify Labs
External co-authors :
no
Language :
English
Title :
Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Space
Publication date :
May 2023
Event name :
5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP
Event organizer :
Association for Computational Linguistics
Event place :
Dubrovnik, Croatia
Event date :
May, 6th 2023
Audience :
International
Main work title :
Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP
Tyler Chang, Zhuowen Tu, and Benjamin Bergen. 2022. The Geometry of Multilingual Language Model Representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 119-136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chris Collins and Richard Kayne. 2011. Syntactic Structures of the World?s Languages. New York University, New York.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440? 8451, Online. Association for Computational Linguistics.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating Crosslingual Sentence Representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
Wietse de Vries, Martijn Wieling, and Malvina Nissim. 2022. Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7676-7685, Dublin, Ireland. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Blazej Dolicki and Gerasimos Spanakis. 2021. Analysing The Impact Of Linguistic Features On Cross-Lingual Transfer. ArXiv:2105.05975 [cs].
Matthew S. Dryer and Martin Haspelmath. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
Hila Gonen, Shauli Ravfogel, Yanai Elazar, and Yoav Goldberg. 2020. It?s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 45-56, Online. Association for Computational Linguistics.
Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2015. Glottolog 2.6. Max Planck Institute for the Science of Human History, Jena.
Ganesh Jawahar, Beno?t Sagot, and Djamé Seddah. 2019. What Does BERT Learn about the Structure of Language In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of Neural Network Representations Revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519-3529. PMLR.
Anne Lauscher, Vinit Ravishankar, Ivan Vulíc, and Goran Glavaš. 2020. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499, Online. Association for Computational Linguistics.
M. Paul Lewis, Gary F. Simons, and Charles D. Fennig. 2015. Ethnologue: Languages of the World, Eighteenth edition. SIL International, Dallas, Texas.
Jindrich Libovický, Rudolf Rosa, and Alexander Fraser. 2020. On the Language Neutrality of Pre-Trained Multilingual Representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663-1674, Online. Association for Computational Linguistics.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8-14, Valencia, Spain. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What Happens To BERT Embeddings During Fine-Tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 33-44, Online. Association for Computational Linguistics.
Steven Moran, Daniel McCloy, and (eds.). 2019. PHOIBLE 2.0. Max Planck Institute for the Science of Human History, Jena.
Benjamin Muller, Yanai Elazar, Beno?t Sagot, and Djamé Seddah. 2021. First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2214-2231, Online. Association for Computational Linguistics.
Yulia Otmakhova, Karin Verspoor, and Jey Han Lau. 2022. Cross-linguistic Comparison of Linguistic Feature Encoding in BERT Models for Typologically Different Languages. In Proceedings of the 4th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 27-35, Seattle, Washington. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is Not an Interlingua and the Bias of Tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47-55, Hong Kong, China. Association for Computational Linguistics.
Marc Tanti, Lonneke van der Plas, Claudia Borg, and Albert Gatt. 2021. On the Language-specificity of Multilingual BERT and the Impact of Fine-Tuning. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 214-227, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-The-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844, Hong Kong, China. Association for Computational Linguistics.
Weijia Xu, Batool Haider, Jason Krone, and Saab Mansour. 2021. Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer. In Proceedings of the 1st Workshop on Meta Learning and Its Applications to Natural Language Processing, pages 11-18, Online. Association for Computational Linguistics.
