Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset

Transformer-based models have shown outstanding results in natural language
processing but face challenges in applications like classifying small-scale
clinical texts, especially with constrained computational resources. This study
presents a customized Mixture of Expert (MoE) Transformer models for
classifying small-scale French clinical texts at CHU Sainte-Justine Hospital.
The MoE-Transformer addresses the dual challenges of effective training with
limited data and low-resource computation suitable for in-house hospital use.
Despite the success of biomedical pre-trained models such as CamemBERT-bio,
DrBERT, and AliBERT, their high computational demands make them impractical for
many clinical settings. Our MoE-Transformer model not only outperforms
DistillBERT, CamemBERT, FlauBERT, and Transformer models on the same dataset
but also achieves impressive results: an accuracy of 87\%, precision of 87\%,
recall of 85\%, and F1-score of 86\%. While the MoE-Transformer does not
surpass the performance of biomedical pre-trained BERT models, it can be
trained at least 190 times faster, offering a viable alternative for settings
with limited data and computational resources. Although the MoE-Transformer
addresses challenges of generalization gaps and sharp minima, demonstrating
some limitations for efficient and accurate clinical text classification, this
model still represents a significant advancement in the field. It is
particularly valuable for classifying small French clinical narratives within
the privacy and constraints of hospital-based computational resources.