强化学习与差分隐私.pdf

1 Introduction

Recent years have witnessed a boom in artificial intelligence, which contributes to

a wide range of applications, such as face recognition, self-driving, medical diag-

nosis, etc. [17, 25]. Notably, the success of AlphaGo

of reinforcement learning. Nevertheless, the security and privacy issues combined

with artificial intelligence also draw full attention from researchers in the mean-

time. In real-world scenarios, trained reinforcement learning policies are released

to client-side and often contain sensitive information, from which adversaries may

2 P. Ma et al.

knowledge), we still think the environment should be protected since it contains

sensitive information.

a strong privacy model which provides formal privacy guarantees that do not

depend on the background knowledge or computational power of an adversary

[11]. Apple Inc. integrated differential privacy for collecting sensitive data in its

operating systems, iOS and macOS, respectively. Google released a framework

for local differentially private data aggregation and deployed it in Chrome [9].

While differential privacy for reinforcement learning in specific cases has been

forcement learning contexts. However, differing from learning models with initial

datasets, as is shown in Fig. 1, reinforcement learning does not have the notion

vacy guarantees. However, today’s reinforcement learning models usually adopt

deep neural networks to approximate action values or policies, and it is a com-

mon belief that these neural networks are hard to be analyzed theoretically. So in

this paper, we illustrate the ideas of the mechanism achieving differential privacy

in a simplified setting, i.e., the K-armed bandit problem, which still preserves

the important features distinguishing reinforcement learning from other types of

learning.

4 presents the mechanism design for -greedy and Softmax, respectively. Section

5 describes our experimental results. Section 6 discusses related work, and Sec-

tion 7 concludes.

2 Preliminaries

In this section, we briefly introduce notions of reinforcement learning and differ-

ential privacy.

2.1 Reinforcement Learning

Reinforcement learning is a set of machine learning methods concerned with

how agents take actions in an environment for maximising cumulative reward

4 P. Ma et al.

[25]. Typically, the reinforcement learning problem can be cast as a Markov

Decision Process (MDP) [13]. For agents in the environment E, the state space

X , where each x ∈ X denotes the stage of an agent in the environment E, when

an action a ∈ A is taken, reward will be given by the environment E based on

the reward function R. To summarise, a quaternion E = hX , A, P, Ri denote

the reinforcement learning model, where P : X × A × X → R denotes the state

transition probability, R : X × A × X → R denotes the reward function of the

environment E.

understood as time passes [12, 3]. Occasionally, the bandit algorithms tend to

be trapped into Exploration-Exploitation dilemma, where the agents strive to

balance sufficiently exploring the variant space and exploiting the optimal action.

axiom and picks an arm with the probability given by Boltzmann distribution

according to its average reward [16]. Following is the p.d.f. of each action.

2.2 Differential Privacy

With the growth of data aggregation and mining, the threats to data privacy

also increase. Roughly speaking, differential privacy is a mathematical model of

data privacy guarantees in a statistical dataset [7, 4–6]. The objective of differ-

ential privacy is to perturb the output of queries to prevent adversaries infer the

demographic information. The noise is controlled by the privacy budget 

] to denote a statistical database, where