Experience Replay

### Experience Replay 概念

Experience Replay 是一种用于提高强化学习算法稳定性和效率的技术。通过存储代理在过去经历过的状态、动作、奖励和下一个状态的元组到一个称为回放缓冲区的数据结构中，在训练过程中可以随机抽取这些经验样本进行更新，而不是仅依赖最近的经验数据[^2]。

这种方法有助于打破数据之间的关联性，使得模型能够更有效地泛化所学的知识。此外，它还允许重复利用历史数据，从而提高了数据使用的效率并减少了对新交互的需求。

### 实现方法

为了实现 Experience Replay 功能，通常会创建一个循环缓冲区来保存一定数量的历史体验样本。当这个缓冲区满了之后，新的样本将会替换掉最旧的那个。下面是一个简单的 Python 版本的 Experience Replay 缓存类：

import random
from collections import deque, namedtuple

class ReplayBuffer:
    """Fixed-size buffer to store experience tuples."""

    def __init__(self, buffer_size, batch_size):
        self.memory = deque(maxlen=buffer_size)  
        self.batch_size = batch_size
        self.experience = namedtuple("Experience", field_names=["state", "action", "reward", "next_state", "done"])

    def add(self, state, action, reward, next_state, done):
        e = self.experience(state, action, reward, next_state, done)
        self.memory.append(e)

    def sample(self):
        experiences = random.sample(self.memory, k=self.batch_size)

        states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)
        actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).long().to(device)
        rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)
        next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(device)
        dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(device)

        return (states, actions, rewards, next_states, dones)

    def __len__(self):
        return len(self.memory)

这段代码定义了一个 `ReplayBuffer` 类，该类维护着一个固定大小的记忆列表，并提供了向其中添加新条目以及从中采样一批条目的功能。注意这里假设使用 PyTorch 进行张量操作；如果采用其他框架，则需相应调整转换逻辑。