RL From Scratch

Must Read

Implementations

• DQNValue-Based

  Deep Q-Network - neural network approximates Q(s,a), stabilised with experience replay and a frozen target network (Mnih et al., 2015).

  DiscreteOff-policyMLP
  →
• Double DQNValue-Based

  Same loop as DQN, one change in the target - the online net picks the action, the target net scores it, so the max stops flattering itself (van Hasselt et al., 2016).

  DiscreteOff-policyMLP
  →
• Q-learningValue-Based

  Tabular off-policy TD control - a Q-table for small discrete Gym envs like FrozenLake and CliffWalking (Watkins, 1989). The foundation DQN later approximates with a net.

  DiscreteOff-policyTabular
  →
• Policy-Based MethodsPolicy Gradient

  Notes on optimising the policy directly - gradient ascent on J(θ), why the environment's transition probability falls out of the gradient, and the policy gradient theorem derived two ways.

  Discrete + Cont.On-policyStochastic
  →
• REINFORCEPolicy Gradient

  Monte Carlo policy gradient - collect full episodes, compute returns, reinforce actions that led to high reward. The original policy gradient method (Williams, 1992).

  DiscreteOn-policyMLP · CNN
  →

About NeatRL

Citation

@misc{singh2026neatrl,
  title={NeatRL: Readable Single-File Reinforcement Learning Implementations in PyTorch},
  author={Yuvraj Singh},
  year={2026},
  howpublished={\url{https://github.com/YuvrajSingh-mist/NeatRL}},
  doi={10.5281/zenodo.23198154},
}