![](https://csdnimg.cn/release/download_crawler_static/88570598/bg1.jpg)
CD
2
-pFed: Cyclic Distillation-guided Channel Decoupling for Model
Personalization in Federated Learning
Yiqing Shen
1
, Yuyin Zhou
2
, Lequan Yu
3
*
1
Shanghai Jiao Tong University,
2
UC Santa Cruz,
3
The University of Hong Kong
shenyq@sjtu.edu.cn, zhouyuyiner@gmail.com, lqyu@hku.hk
Abstract
Federated learning (FL) is a distributed learning
paradigm that enables multiple clients to collaboratively
learn a shared global model. Despite the recent progress,
it remains challenging to deal with heterogeneous data
clients, as the discrepant data distributions usually pre-
vent the global model from delivering good generalization
ability on each participating client. In this paper, we pro-
pose CD
2
-pFed, a novel Cyclic Distillation-guided Channel
Decoupling framework, to personalize the global model in
FL, under various settings of data heterogeneity. Differ-
ent from previous works which establish layer-wise per-
sonalization to overcome the non-IID data across different
clients, we make the first attempt at channel-wise assign-
ment for model personalization, referred to as channel de-
coupling. To further facilitate the collaboration between
private and shared weights, we propose a novel cyclic dis-
tillation scheme to impose a consistent regularization be-
tween the local and global model representations during the
federation. Guided by the cyclical distillation, our chan-
nel decoupling framework can deliver more accurate and
generalized results for different kinds of heterogeneity, such
as feature skew, label distribution skew, and concept shift.
Comprehensive experiments on four benchmarks, including
natural image and medical image analysis tasks, demon-
strate the consistent effectiveness of our method on both lo-
cal and external validations.
1. Introduction
Deep learning techniques have received notable attention
in various vision tasks, such as image classification [7], ob-
ject detection [31], and semantic segmentation [25]. Yet,
the success of deep neural networks heavily relies on a
tremendous volume of valuable training images. One possi-
ble solution is to collaboratively curate numerous data sam-
ples from different parties (e.g., different mobile devices
*
Corresponding Author.
Figure 1. Illustration of different parameter decoupling manners
for model personalization in Federated Learning. The previous
approaches combine local and global parameters in a layer-wise
mechanism, including LG-Fed [22] in low-level input layers (a)
and FedPer [2] in high-level output layers (b). Instead, we achieve
model personalization via channel-wise decoupling (c).
and companies). However, collecting distributed data into a
centralized storage facility is costly and time-consuming.
Additionally, in real practice, decentralized image data
should not be directly shared, due to privacy concerns or
legal restrictions [1, 39]. In this case, conventional central-
ized machine learning frameworks fail to satisfy the data
privacy protection constraint.
Therefore, the data-private distributed training
paradigms, especially Federated Learning (FL), have
received an increasing popularity [3–5,19,24,28,36,47,50].
To be more specific, in FL, a shared model is globally
trained with an orchestration of local updates within data
stored at each client. A pioneering FL algorithm named
Federated Average (FedAVG), aggregates parameters at the
central server by communication across clients once per
global epoch, without explicit data sharing [28]. Compared
with local training, the federation on a larger scale of
training data has demonstrated its superiority to boost the
generalization ability on unseen data, with the orchestration
of distributed private data [6, 28].
10041