联邦学习AdvancesandOpenProblemsinFederatedLearning_联合学习

需积分: 48 626 浏览量更新于2023-03-03 评论收藏 1.23MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

Advances and Open Problems in Federated Learning

Peter Kairouz

7∗

H. Brendan McMahan

7∗

Brendan Avent

Aur

elien Bellet

Mehdi Bennis

Arjun Nitin Bhagoji

Keith Bonawitz

Zachary Charles

Graham Cormode

Rachel Cummings

Rafael G.L. D’Oliveira

Salim El Rouayheb

David Evans

Josh Gardner

Zachary Garrett

Adri

a Gasc

Badih Ghazi

Phillip B. Gibbons

Marco Gruteser

7,14

Zaid Harchaoui

Chaoyang He

Lie He

Zhouyuan Huo

Ben Hutchinson

Justin Hsu

Martin Jaggi

Tara Javidi

Gauri Joshi

Mikhail Khodak

Jakub Kone

Aleksandra Korolova

Farinaz Koushanfar

Sanmi Koyejo

7,18

Tancr

ede Lepoint

Yang Liu

Prateek Mittal

Mehryar Mohri

Richard Nock

Ayfer

Ozg

Rasmus Pagh

7,10

Mariana Raykova

Hang Qi

Daniel Ramage

Ramesh Raskar

Dawn Song

Weikang Song

Sebastian U. Stich

Ziteng Sun

Ananda Theertha Suresh

Florian Tram

Praneeth Vepakomma

Jianyu Wang

Li Xiong

Zheng Xu

Qiang Yang

Felix X. Yu

Han Yu

Sen Zhao

Australian National University,

Carnegie Mellon University,

Cornell University,

Ecole Polytechnique F

erale de Lausanne,

Emory University,

Georgia Institute of Technology,

Google Research,

Hong Kong University of Science and Technology,

INRIA,

IT University of Copenhagen,

Massachusetts Institute of Technology,

Nanyang Technological University,

Princeton University,

Rutgers University,

Stanford University,

University of California Berkeley,

University of California San Diego,

University of Illinois Urbana-Champaign,

University of Oulu,

University of Pittsburgh,

University of Southern California,

University of Virginia,

University of Warwick,

University of Washington,

University of Wisconsin–Madison

Abstract

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or

whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service

provider), while keeping the training data decentralized. FL embodies the principles of focused data

collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting

from traditional, centralized machine learning and data science approaches. Motivated by the explosive

growth in FL research, this paper discusses recent advances and presents an extensive collection of open

problems and challenges.

∗

Peter Kairouz and H. Brendan McMahan conceived, coordinated, and edited this work. Correspondence to kairouz@

google.com and mcmahan@google.com.

arXiv:1912.04977v1 [cs.LG] 10 Dec 2019

Contents

1 Introduction 4

1.1 The Cross-Device Federated Learning Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1.1 The Lifecycle of a Model in Federated Learning . . . . . . . . . . . . . . . . . . . . . . . . 7

1.1.2 A Typical Federated Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.2 Federated Learning Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.3 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 Relaxing the Core FL Assumptions: Applications to Emerging Settings and Scenarios 11

2.1 Fully Decentralized / Peer-to-Peer Distributed Learning . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1 Algorithmic Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.1.2 Practical Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2 Cross-Silo Federated Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.3 Split Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Improving Efﬁciency and Effectiveness 18

3.1 Non-IID Data in Federated Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1.1 Strategies for Dealing with Non-IID Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.2 Optimization Algorithms for Federated Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.2.1 Optimization Algorithms and Convergence Rates for IID Datasets . . . . . . . . . . . . . . . 21

3.2.2 Optimization Algorithms and Convergence Rates for Non-IID Datasets . . . . . . . . . . . . 25

3.3 Multi-Task Learning, Personalization, and Meta-Learning . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3.1 Personalization via Featurization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3.2 Multi-Task Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.3.3 Local Fine Tuning and Meta-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3.4 When is a Global FL-trained Model Better? . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.4 Adapting ML Workﬂows for Federated Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.4.1 Hyperparameter Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.4.2 Neural Architecture Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.4.3 Debugging and Interpretability for FL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.5 Communication and Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.6 Application To More Types of Machine Learning Problems and Models . . . . . . . . . . . . . . . . 34

4 Preserving the Privacy of User Data 35

4.1 Actors, Threat Models, and Privacy in Depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.2 Tools and Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4.2.1 Secure Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.2.2 Privacy-Preserving Disclosures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.2.3 Veriﬁability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.3 Protections Against External Malicious Actors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.3.1 Auditing the Iterates and Final Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.3.2 Training with Central Differential Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.3.3 Concealing the Iterates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.3.4 Repeated Analyses over Evolving Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.3.5 Preventing Model Theft and Misuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4.4 Protections Against an Adversarial Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.4.1 Challenges: Communication Channels, Sybil Attacks, and Selection . . . . . . . . . . . . . . 52

4.4.2 Limitations of Existing Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

4.4.3 Training with Distributed Differential Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4.4.4 Preserving Privacy While Training Sub-Models . . . . . . . . . . . . . . . . . . . . . . . . . 56

4.5 User Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.5.1 Understanding Privacy Needs for Particular Analysis Tasks . . . . . . . . . . . . . . . . . . . 58

1 Introduction

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole or-

ganizations) collaboratively train a model under the orchestration of a central server (e.g. service provider),

while keeping the training data decentralized. It embodies the principles of focused collection and data

minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, cen-

tralized machine learning. This area has received signiﬁcant interest recently, both from research and applied

perspectives. This paper describes the deﬁning characteristics and challenges of the federated learning set-

ting, highlights important practical constraints and considerations, and then enumerates a range of valuable

research directions. The goals of this work are to highlight research problems that are of signiﬁcant theo-

retical and practical interest, and to encourage research on problems that could have signiﬁcant real-world

impact.

The term federated learning was introduced in 2016 by McMahan et al. [289]: “We term our approach

Federated Learning, since the learning task is solved by a loose federation of participating devices (which

we refer to as clients) which are coordinated by a central server.” An unbalanced and non-IID (identically

and independently distributed) data partitioning across a massive number of unreliable devices with limited

communication bandwidth was introduced as the deﬁning set of challenges.

Signiﬁcant related work predates the introduction of the term federated learning. A longstanding goal

pursued by many research communities (including cryptography, databases, and machine learning) is to ana-

lyze and learn from data distributed among many owners without exposing that data. Cryptographic methods

for computing on encrypted data were developed starting in the early 1980s [340, 421], and Agrawal and

Srikant [15] and Vaidya et al. [390] are early examples of work that sought to learn from local data using

a centralized server while preserving privacy. Conversely, even since the introduction of the term federated

learning, we are aware of no single work that directly addresses the full set of FL challenges. Thus, the term

federated learning provides a convenient shorthand for a set of characteristics, constraints, and challenges

that often co-occur in applied ML problems on decentralized data where privacy is paramount.

This paper originated at the Workshop on Federated Learning and Analytics held June 17–18th, 2019,

hosted at Google’s Seattle ofﬁce. During the course of this two-day event, the need for a broad paper

surveying the many open challenges in the area of federated learning became clear.

A key property of many of the problems discussed is that they are inherently interdisciplinary — solving

them likely requires not just machine learning, but techniques from distributed optimization, cryptography,

security, differential privacy, fairness, compressed sensing, systems, information theory, statistics, and more.

Many of the hardest problems are at the intersections of these areas, and so we believe collaboration will be

essential to ongoing progress. One of the goals of this work is to highlight the ways in which techniques from

these ﬁelds can potentially be combined, raising both interesting possibilities as well as new challenges.

Since the term federated learning was initially introduced with an emphasis on mobile and edge device

applications [289, 287], interest in applying FL to other applications has greatly increased, including some

which might involve only a small number of relatively reliable clients, for example multiple organizations

collaborating to train a model. We term these two federated learning settings “cross-device” and “cross-silo”

respectively. Given these variations, we propose a somewhat broader deﬁnition of federated learning:

Federated learning is a machine learning setting where multiple entities (clients) collaborate

in solving a machine learning problem, under the coordination of a central server or service

provider. Each client’s raw data is stored locally and not exchanged or transferred; instead,

During the preparation of this work, Li et al. [262] independently released an excellent but less comprehensive survey.

focused updates intended for immediate aggregation are used to achieve the learning objective.

Focused updates are updates narrowly scoped to contain the minimum information necessary for the speciﬁc

learning task at hand; aggregation is performed as earlier as possible in the service of data minimization.

We note that this deﬁnition distinguishes federated learning from fully decentralized (peer-to-peer) learning

techniques as discussed in Section 2.1.

Although privacy-preserving data analysis has been studied for more than 50 years, only in the past

decade have solutions been widely deployed at scale (e.g. [156, 135]). Cross-device federated learning and

federated data analysis are now being applied in consumer digital products. Google makes extensive use of

federated learning in the Gboard mobile keyboard [323, 196, 420, 98, 329], as well as in features on Pixel

phones [18] and in Android Messages [375]. While Google has pioneered cross-device FL, interest in this

setting is now much broader, for example: Apple is using cross-device FL in iOS 13 [27], for applications

like the QuickType keyboard and the vocal classiﬁer for “Hey Siri” [28]; doc.ai is developing cross-device

FL solutions for medical research [130], and Snips has explored cross-device FL for hotword detection

[259].

Cross-silo applications have also been proposed or described in myriad domains including ﬁnance risk

prediction for reinsurance [407], pharmaceuticals discovery [158], electronic health records mining [162],

medical data segmentation [19, 121], and smart manufacturing [305].

The growing demand for federated learning technology has resulted in a number of tools and frameworks

becoming available. These include TensorFlow Federated [38], Federated AI Technology Enabler [34],

PySyft [342], Leaf [35], PaddleFL [36] and Clara Training Framework [33]; more details in Appendix A.

Commercial data platforms incorporating federated learning are in development from established technology

companies as well as smaller start-ups.

Table 1 contrasts both cross-device and cross-silo federated learning with traditional single-datacenter

distributed learning across a range of axes. These characteristics establish many of the constraints that

practical federated learning systems must typically satisfy, and hence serve to both motivate and inform the

open challenges in federated learning. They will be discussed at length in the sections that follow.

These two FL variants are called out as representative and important examples, but different FL settings

may have different combinations of these characteristics. For the remainder of this paper, we consider the

cross-device FL setting unless otherwise noted, though many of the problems apply to other FL settings as

well. Section 2 speciﬁcally addresses some of the many other variations and applications.

Next, we consider cross-device federated learning in more detail, focusing on practical aspects common

to a typical large-scale deployment of the technology; Bonawitz et al. [74] provides even more detail for a

particular production system, including a discussion of speciﬁc architectural choices and considerations.

1.1 The Cross-Device Federated Learning Setting

This section takes an applied perspective, and unlike the previous section, does not attempt to be deﬁnitional.

Rather, the goal is to describe some of the practical issues in cross-device FL and how they might ﬁt into a

broader machine learning development and deployment ecosystem. The hope is to provide useful context

and motivation for the open problems that follow, as well as to aid researchers in estimating how straight-

forward it would be to deploy a particular new approach in a real-world system. We begin by sketching the

lifecycle of a model before considering a FL training process.

剩余104页未读，继续阅读

5665dfg786769

粉丝: 27
资源: 15

会员权益专享

联邦学习Advances and Open Problems in Federated Learning

评论0

会员权益专享

最新资源

联邦学习Advances and Open Problems in Federated Learning

评论0

advances and open problems in federated learning

advances in financial machine learning

recent advances in deep learning for object detection

Advances in Audio and Speech Signal Processing

deep multimodal learning a survey on recent advances and trends

Introduction.to.Deep.Learning.with.complete.Python.and.TensorFlow.examples

Advances in Artificial Intelligence 无水印原版pdf

给出近年的自主机器学习的参考文献

人工智能深度学习参考文献

advances in neural information processing systems

有没有近两年的NER综述

Propagation of Sound in Porous Media

ieee icassp recent advances in nonnegative matrix factorization

C# 6.0 and the .NET 4.6 Framework(7th).pdf 2016第7版pdf

给出分布式深度神经网络参考文献

会员权益专享

最新资源