Hands-On Generative AI with Transformers and Diffusion Models
By Omar Sanseviero, Pedro Cuenca, Apolinário Passos and Jonathan Whitaker
Synopsis
Published by O'Reilly in late 2024, this book explains how modern generative models for text, images and audio work and how to adapt them with open-source tools. It opens with an overview of generative media, then covers transformers, the compression and representation of information with autoencoders and embeddings, and diffusion models. Later chapters treat Stable Diffusion and conditional generation, fine-tuning language models, fine-tuning Stable Diffusion, creative applications of text-to-image models such as image editing, and audio generation, with an appendix on building an end-to-end retrieval-augmented generation pipeline.
Each chapter pairs a conceptual explanation with runnable code, which the authors publish in a GitHub repository with Google Colab links, and the examples rely on Hugging Face libraries such as transformers and diffusers. Exercises ask readers to extend models rather than simply rerun them. The authors discuss when a pretrained model suffices and when fine-tuning, including parameter-efficient methods such as LoRA, earns its cost. Readers gain working fluency with the open-model ecosystem and enough theory to follow new research papers, though those focused on proprietary frontier models or production deployment will need other sources.
Scorecard
Editorial score
How it scored
Editorial score: 69.8 / 1005 weighted criteria
- 15.0/30
Influence30%
- 20.0/25
Authority25%
- 16.0/20
Practical value20%
- 11.3/15
Currency15%
- 7.5/10
Readability10%
- 69.8/100
Editorial score
How the rank was computed
Rank No. 229 of 500 comes from a composite of 55.6: observable signals 24.1 of 55 points plus editorial review 31.5 of 45 points.
All 11 inputs
| Input | Points |
|---|---|
| Observable signals | 24.1/55 |
| Reach (Amazon ratings volume) | 5.4/16.5 |
| Consensus across best-of lists | 11.0/13.8 |
| Awards and bestseller evidence | 0.0/8.3 |
| Reader rating, sample-adjusted | 5.1/8.3 |
| Amazon sales rank | 0.9/5.5 |
| Recency | 1.7/2.8 |
| Editorial review | 31.5/45 |
| Author authority | 9.0/11.3 |
| Influence | 6.8/13.5 |
| Practical value | 7.2/9.0 |
| Readability | 3.4/4.5 |
| Currency | 5.1/6.8 |
Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.
Verdict
A clear, code-rich guide to how open generative models work and how to fine-tune them, written largely by Hugging Face practitioners.
Strengths
- Runnable notebooks with Colab links
- Covers text, image and audio generation
- Authors work on the tools they teach
Keep in mind
It assumes Python and some deep learning background, needs GPU access for many examples, and gives limited attention to production deployment.
Key takeaways
- Transformers and diffusion models share common building blocks, and understanding tokenization, embeddings and attention makes both families easier to adapt.
- Pretrained open models cover many tasks; fine-tuning, often with parameter-efficient methods such as LoRA, pays off when data or style needs are specific.
- Diffusion models generate images by learning to reverse a gradual noising process, and conditioning signals such as text prompts steer the output.
- Combining existing components, such as text encoders, latent autoencoders and noise schedulers, can produce new creative applications without training from scratch.
Notable facts
- Companion code for all nine chapters, plus an appendix on end-to-end retrieval-augmented generation, is published in the genaibook GitHub repository with Colab links. Source (opens in a new tab)
About the author
Questions
What do I need to know before reading Hands-On Generative AI with Transformers and Diffusion Models?
Does the book cover large language models like ChatGPT?
Is Hands-On Generative AI worth reading in 2026?
Related books
- Generative AI Design Patterns Valliappa Lakshmanan and Hannes Hapke
- Reinforcement Learning from Human Feedback Nathan Lambert
- Agentic Artificial Intelligence Pascal Bornet, Jochen Wirtz, Thomas H. Davenport, David De Cremer, Brian Evergreen, Phil Fersht, Rakesh Gohel and Shail Khiyara
- Agentic Design Patterns Antonio Gullí
- Building Applications with AI Agents Michael Albada
- Principles of Building AI Agents Sam Bhagwat
Sources
As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.






