Paper published in a book (Scientific congresses, symposiums and conference proceedings)
LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish
LOTHRITZ, Cedric; LEBICHOT, Bertrand; ALLIX, Kevin et al.
2022In Proceedings of the Language Resources and Evaluation Conference, 2022
Peer reviewed
 

Files


Full Text
LuxemBERT_LREC.pdf
Author postprint (642.82 kB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
language modelling; natural language processing; data augmentation
Abstract :
[en] Pre-trained Language Models such as BERT have become ubiquitous in NLP where they have achieved state-of-the-art performance in most NLP tasks. While these models are readily available for English and other widely spoken languages, they remain scarce for low-resource languages such as Luxembourgish. In this paper, we present LuxemBERT, a BERT model for the Luxembourgish language that we create using the following approach: we augment the pre-training dataset by considering text data from a closely related language that we partially translate using a simple and straightforward method. We are then able to produce the LuxemBERT model, which we show to be effective for various NLP tasks: it outperforms a simple baseline built with the available Luxembourgish text data as well the multilingual mBERT model, which is currently the only option for transformer-based language models in Luxembourgish. Furthermore, we present datasets for various downstream NLP tasks that we created for this study and will make available to researchers on request.
Disciplines :
Computer science
Author, co-author :
LOTHRITZ, Cedric  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
LEBICHOT, Bertrand ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
ALLIX, Kevin ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
VEIBER, Lisa ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
BISSYANDE, Tegawendé François D Assise  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
KLEIN, Jacques  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX
Boytsov, Andrey;  Banque BGL BNP Paribas
Goujon, Anne;  Banque BGL BNP Paribas
Lefebvre, Clément;  Banque BGL BNP Paribas
External co-authors :
no
Language :
English
Title :
LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish
Publication date :
June 2022
Event name :
13th Language Resources and Evaluation Conference (LREC 2022)
Event organizer :
European Language Resources Association
Event place :
Marseille, France
Event date :
20.06.2022-25.06.2022
Audience :
International
Main work title :
Proceedings of the Language Resources and Evaluation Conference, 2022
Pages :
5080-5089
Peer reviewed :
Peer reviewed
Focus Area :
Computational Sciences
Available on ORBilu :
since 01 August 2022

Statistics


Number of views
635 (91 by Unilu)
Number of downloads
319 (46 by Unilu)

Scopus citations®
 
6
Scopus citations®
without self-citations
5

Bibliography


Similar publications



Contact ORBilu