Multi-objective Representation for Numbers in Clinical Narratives Using CamemBERT-bio

This research aims to classify numerical values extracted from medical
documents across seven distinct physiological categories, employing
CamemBERT-bio. Previous studies suggested that transformer-based models might
not perform as well as traditional NLP models in such tasks. To enhance
CamemBERT-bio's performances, we introduce two main innovations: integrating
keyword embeddings into the model and adopting a number-agnostic strategy by
excluding all numerical data from the text. The implementation of label
embedding techniques refines the attention mechanisms, while the technique of
using a `numerical-blind' dataset aims to bolster context-centric learning.
Another key component of our research is determining the criticality of
extracted numerical data. To achieve this, we utilized a simple approach that
involves verifying if the value falls within the established standard ranges.
Our findings are encouraging, showing substantial improvements in the
effectiveness of CamemBERT-bio, surpassing conventional methods with an F1
score of 0.89. This represents an over 20\% increase over the 0.73 $F_1$ score
of traditional approaches and an over 9\% increase over the 0.82 $F_1$ score of
state-of-the-art approaches. All this was achieved despite using small and
imbalanced training datasets.