• Computer Science > Distributed, Parallel, and Cluster Computing [Submitted on 24 Jan 2025 (v1), last revised 18 Feb 2026 (this version, v4)] Title:Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models View PDF HTML (experimental)Abstract:Pre-trained Language Models (PLMs) have demonstrated their superiority and versatility in modern Natural Language Processing (NLP), effectively adapting to various downstream tasks through further fine-tuning. • Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a promising solution to address privacy and efficiency challenges in distributed training for PLMs on resource-constrained local devices. • However, our measurements reveal two key limitations of FedPEFT: heterogeneous data across devices exacerbates performance degradation of low-rank adaptation, and a fixed parameter configuration results in communication inefficiency. • To overcome these limitations, we propose FedARA, a novel adaptive rank allocation framework for federated parameter-efficient fine-tuning of language models. • Specifically, FedARA employs truncated Singular Value Decomposition (SVD) adaptation to enhance similar feature representation across clients, significantly mitigating the adverse effects of data heterogeneity. • Subsequently, it utilizes dynamic rank allocation to progressively identify critical ranks, effectively improving communication efficiency.
Article Summaries:
- Adaptive Rank Allocation for Federated Parameter‑Efficient Fine‑Tuning of Language Models Researchers have introduced FedARA, a framework that improves federated fine‑tuning of pre‑trained language models on edge devices. FedARA first applies truncated singular‑value decomposition to align feature representations across heterogeneous client data, mitigating performance drops seen in low‑rank adaptation. It then dynamically allocates ranks, progressively identifying critical components to reduce communication overhead, and prunes inactive modules to lower local compute and memory use. Experiments show FedARA outperforms existing baselines by 6.95-8.49 % on diverse datasets, boosts communication efficiency by 2.40×, and cuts total training time and energy consumption by up to 48.9 % and 47.0 %, respectively.
- The paper introduces FedARA, an adaptive rank‑allocation framework for federated parameter‑efficient fine‑tuning (FedPEFT) of language models. It addresses two key FedPEFT shortcomings: performance drops from heterogeneous client data and static rank settings that waste communication bandwidth. FedARA applies truncated singular‑value decomposition to align feature representations across clients, then dynamically selects critical low‑rank components to reduce communication. It also prunes inactive modules, cutting local compute and memory use. Experiments on diverse datasets and models show 6.9-8.5 % accuracy gains over baselines, a 2.4× communication speed‑up, and up to 49 % reductions in training time and 47 % in energy on edge devices.
Sources: