Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates

Continual learning (CL) refers to the ability of an intelligent system to
sequentially acquire and retain knowledge from a stream of data with as little
computational overhead as possible. To this end; regularization, replay,
architecture, and parameter isolation approaches were introduced to the
literature. Parameter isolation using a sparse network which enables to
allocate distinct parts of the neural network to different tasks and also
allows to share of parameters between tasks if they are similar. Dynamic Sparse
Training (DST) is a prominent way to find these sparse networks and isolate
them for each task. This paper is the first empirical study investigating the
effect of different DST components under the CL paradigm to fill a critical
research gap and shed light on the optimal configuration of DST for CL if it
exists. Therefore, we perform a comprehensive study in which we investigate
various DST components to find the best topology per task on well-known
CIFAR100 and miniImageNet benchmarks in a task-incremental CL setup since our
primary focus is to evaluate the performance of various DST criteria, rather
than the process of mask selection. We found that, at a low sparsity level,
Erdos-R\'enyi Kernel (ERK) initialization utilizes the backbone more
efficiently and allows to effectively learn increments of tasks. At a high
sparsity level, unless it is extreme, uniform initialization demonstrates a
more reliable and robust performance. In terms of growth strategy; performance
is dependent on the defined initialization strategy and the extent of sparsity.
Finally, adaptivity within DST components is a promising way for better
continual learners.