Federated Geometric Monte Carlo Clustering to Counter Non-IID Datasets

Federated learning allows clients to collaboratively
train models on datasets that are acquired in different locations
and that cannot be exchanged because of their size or regulations.
Such collected data is increasingly non-independent and non-
identically distributed (non-IID), negatively affecting training
accuracy. Previous works tried to mitigate the effects of non-
IID datasets on training accuracy, focusing mainly on non-IID
labels, however practical datasets often also contain non-IID
features. To address both non-IID labels and features, we propose
FedGMCC1, a novel framework where a central server aggregates
client models that it can cluster together. FedGMCC clustering relies
on a Monte Carlo procedure that samples the output space of
client models, infers their position in the weight space on a loss
manifold and computes their geometric connection via an affine
curve parametrization. FedGMCC aggregates connected models
along their path connectivity to produce a richer global model,
incorporating knowledge of all connected client models. FedGMCC
outperforms FedAvg and FedProx in terms of convergence rates
on the EMNIST62 and a genomic sequence classification datasets
(by up to +63%). FedGMCC yields an improved accuracy (+4%)
on the genomic dataset with respect to CFL, in high non-IID
feature space settings and label incongruency.