MATLAB实现DDPG

DDPG（深度确定性策略梯度）是一种用于连续动作空间的强化学习算法，在机器人控制、自动驾驶等领域有广泛的应用。下面是MATLAB实现DDPG算法的示例代码：

1. 定义神经网络

% Actor网络
actor = [
imageInputLayer([numStates 1 1],'Normalization','none','Name','state')
fullyConnectedLayer(256,'Name','fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(128,'Name','fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(numActions,'Name','out')
tanhLayer('Name','tanh1')
scalingLayer('Name','actorOutput')
];

% Critic网络
critic = [
imageInputLayer([numStates 1 1],'Normalization','none','Name','state')
fullyConnectedLayer(256,'Name','fc1')
reluLayer('Name','relu1')
fullyConnectedLayer(128,'Name','fc2')
reluLayer('Name','relu2')
fullyConnectedLayer(numActions,'Name','out')
];

2. 定义DDPG算法参数

% 状态空间大小
numStates = 4;

% 动作空间大小
numActions = 2;

% DDPG算法参数
agentOptions = rlDDPGAgentOptions(...
'SampleTime',0.01,...
'TargetSmoothFactor',1e-3,...
'ExperienceBufferLength',1e6,...
'DiscountFactor',0.99,...
'MiniBatchSize',64);

% Actor网络和Critic网络的学习率
actorLearningRate = 1e-4;
criticLearningRate = 1e-3;

% Actor网络和Critic网络的优化器
actorOptimizer = rlRepresentationOptions('Optimizer','adam','LearnRate',actorLearningRate);
criticOptimizer = rlRepresentationOptions('Optimizer','adam','LearnRate',criticLearningRate);

3. 定义环境

% 创建CartPole环境
env = rlPredefinedEnv('CartPole-Continuous');

% 状态空间
observationInfo = env.getObservationInfo();

% 动作空间
actionInfo = env.getActionInfo();

4. 训练DDPG算法

% 创建Actor网络和Critic网络
actorNet = rlFunctionApproximation(actor);
criticNet = rlFunctionApproximation(critic);

% 创建DDPG算法代理
agent = rlDDPGAgent(actorNet,criticNet,agentOptions);

% 训练DDPG算法
maxEpisodes = 1000;
maxSteps = 500;
trainOpts = rlTrainingOptions(...
'MaxEpisodes',maxEpisodes,...
'MaxStepsPerEpisode',maxSteps,...
'ScoreAveragingWindowLength',5,...
'Verbose',false,...
'Plots','training-progress',...
'StopTrainingCriteria','AverageReward',...
'StopTrainingValue',475);
trainingStats = train(agent,env,trainOpts);

5. 测试DDPG算法

% 测试DDPG算法
maxSteps = 500;
simOpts = rlSimulationOptions('MaxSteps',maxSteps);
experience = sim(env,agent,simOpts);

% 绘制CartPole的状态和动作
plot(experience.Observation(:,1))
hold on
plot(experience.Observation(:,2))
plot(experience.Observation(:,3))
plot(experience.Observation(:,4))
plot(experience.Action(:,1))
plot(experience.Action(:,2))
legend('x','x_dot','theta','theta_dot','force')
xlabel('Time Step')
ylabel('State/Action')