Academic CS in Production Code

Prioritized Experience Replay

Tom Schaul, John Quan, Ioannis Antonoglou, David Silver
2015
5 references

Abstract

Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.

View Paper PDF
🤖 Machine Learning
1 repository
5 references

Code References

â–¶ ray-project/ray
4 files
â–¶ doc/source/rllib/rllib-replay-buffers.rst
2
A more advanced strategy (proven better in many cases) is `Prioritized Experiences Replay (PER) <https://arxiv.org/abs/1511.05952>`__.
The following is `from RLlib's examples section <https://github.com/ray-project/ray/blob/master/rllib/examples/replay_buffer_api.py>`__: and runs the R2D2 algorithm with `PER <https://arxiv.org/abs/1511.05952>`__ (which by default it doesn't).
â–¶ rllib/utils/replay_buffers/multi_agent_prioritized_episode_buffer.py
1
batch. See for prioritized sampling Schaul et al. (2016).
â–¶ rllib/utils/replay_buffers/prioritized_episode_buffer.py
1
in the paper by Schaul et al. (2016, https://arxiv.org/abs/1511.05952).
â–¶ rllib/utils/replay_buffers/prioritized_replay_buffer.py
1
Experience Replay". See https://arxiv.org/pdf/1511.05952.pdf for
Link copied to clipboard!