Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution

Large language models (LLMs) have achieved remarkable success across various
tasks but face deployment challenges due to their massive computational demands.
While post-training pruning methods like SparseGPT and Wanda can effectively
reduce the model size, but struggle to maintain model performance at high sparsity
levels, limiting their utility for downstream tasks. Existing fine-tuning methods,
such as full fine-tuning and LoRA, fail to preserve sparsity as they require updating
the whole dense metrics, not well-suited for sparse LLMs. In this paper, we propose
Sparsity Evolution Fine-Tuning (SEFT), a novel method designed specifically
for sparse LLMs. SEFT dynamically evolves the sparse topology of pruned models
during fine-tuning, while preserving the overall sparsity throughout the process.
The strengths of SEFT lie in its ability to perform task-specific adaptation through
a weight drop-and-grow strategy, enabling the pruned model to self-adapt its sparse
connectivity pattern based on the target dataset. Furthermore, a sensitivity-driven
pruning criterion is employed to ensure that the desired sparsity level is consistently
maintained throughout fine-tuning. Our experiments on various LLMs, including
LLaMA families, DeepSeek, and Mistral, across a diverse set of benchmarks
demonstrate that SEFT achieves stronger performance while offering superior
memory and time efficiency compared to existing baselines. Our code is publicly
available at: https://github.com/QiaoXiao7282/SEFT.