No. 335 of 500 No. 65 in Machine Learning

Grokking Deep Reinforcement Learning

By Miguel Morales

Print length
472 pages
Publisher
Manning
Reading level
Intermediate
ISBN-13
978-1617295454
Published
November 10, 2020
Written for
Python developers and ML students who want to understand and implement deep reinforcement learning algorithms.

Synopsis

Miguel Morales, who works on reinforcement learning at Lockheed Martin and teaches in Georgia Tech's Reinforcement Learning and Decision Making course, builds deep reinforcement learning from first principles. The first half develops intuition with tabular methods: Markov decision processes, policy and value iteration, multi-armed bandits and the exploration problem, Monte Carlo and temporal-difference prediction, and control algorithms such as SARSA and Q-learning. The second half adds neural networks, moving from neural fitted Q-iteration through DQN, double DQN, dueling networks and prioritized experience replay, and then to policy-gradient and actor-critic methods including REINFORCE, A3C, GAE, DDPG, TD3, SAC and PPO.

Each algorithm is presented three ways: an intuitive explanation, annotated mathematics, and PyTorch code in Jupyter notebooks from a companion GitHub repository that runs inside a Docker container. A closing chapter surveys topics the book leaves out, such as model-based methods, meta-learning and multi-agent learning, and frames them as steps toward artificial general intelligence. Charles Isbell, then dean of computing at Georgia Tech, wrote the foreword. Readers finish able to implement, tune and debug standard agents and to read current reinforcement learning papers, although the 2020 text does not cover reinforcement learning from human feedback for language models.

Scorecard

Editorial score

How it scored

Editorial score: 65.3 / 1005 weighted criteria

  • Influence30%

    16.5/30
  • Authority25%

    16.3/25
  • Practical value20%

    16.0/20
  • Currency15%

    9.0/15
  • Readability10%

    7.5/10
  • Editorial score

    65.3/100

How the rank was computed

Rank No. 335 of 500 comes from a composite of 51.9: observable signals 22.5 of 55 points plus editorial review 29.4 of 45 points.

All 11 inputs
InputPoints
Observable signals22.5/55
Reach (Amazon ratings volume)7.3/16.5
Consensus across best-of lists8.3/13.8
Awards and bestseller evidence0.0/8.3
Reader rating, sample-adjusted6.2/8.3
Amazon sales rank0.7/5.5
Recency0.0/2.8
Editorial review29.4/45
Author authority7.3/11.3
Influence7.4/13.5
Practical value7.2/9.0
Readability3.4/4.5
Currency4.1/6.8

Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.

Verdict

A patient, code-first introduction to deep reinforcement learning that builds from bandits and MDPs to PPO and soft actor-critic.

Strengths

  • Builds intuition before introducing the mathematics
  • Runnable notebooks for nearly every chapter
  • Covers modern algorithms through SAC and PPO

Keep in mind

Published in 2020, it predates RLHF and large-model applications, and readers need working Python and basic deep learning experience to keep pace.

Key takeaways

  1. Reinforcement learning is hard because feedback is sequential, evaluative and sampled at once, unlike the immediate, labeled feedback of supervised learning.
  2. Exploration has a measurable cost: bandit strategies such as epsilon-greedy, softmax and upper confidence bounds trade information gathering against immediate reward.
  3. DQN stabilizes value-based learning with target networks and experience replay, and double DQN reduces the overestimation of action values.
  4. Actor-critic methods such as SAC and PPO pair a learned policy with value estimates to handle continuous actions and steadier training.

Notable facts

About the author

Miguel Morales

Miguel Morales works on reinforcement learning at Lockheed Martin and is an instructor for the Georgia Institute of Technology's Reinforcement Learning and Decision Making course. He maintains the book's open-source companion repository of chapter notebooks, which provides a Docker environment for running the examples. The foreword is by Charles Isbell, then dean of Georgia Tech's College of Computing.

Questions

What does Grokking Deep Reinforcement Learning cover?
The book moves from reinforcement learning foundations, including Markov decision processes, bandits, Monte Carlo and temporal-difference methods, SARSA and Q-learning, to deep methods such as DQN, double DQN, dueling networks, prioritized replay, REINFORCE, A3C, DDPG, TD3, SAC and PPO. Each algorithm comes with intuitive explanations, annotated math and runnable notebook code, and a final chapter surveys advanced topics toward general intelligence.
Do I need a strong math background for Grokking Deep Reinforcement Learning?
No advanced math is required, but readers should be comfortable with Python and have basic deep learning experience, which Manning lists as a prerequisite. The book introduces equations gradually and annotates them in plain language, so readers with algebra and some probability can keep pace with the derivations. Those wanting rigorous proofs should pair it with Sutton and Barto's Reinforcement Learning: An Introduction.
Is Grokking Deep Reinforcement Learning still useful in 2026?
Yes for its core purpose. The algorithms it teaches, including DQN, PPO and SAC, remain standard baselines in robotics, games and control research, and PPO has been widely used to fine-tune language models. The book predates the spread of reinforcement learning from human feedback and recent reasoning-model training, so readers interested in those topics will need newer sources.

Sources

As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.