No. 305 of 500 No. 43 in Hands-On AI & Programming

Kafka

Real-Time Data and Stream Processing at Scale

By Neha Narkhede, Gwen Shapira and Todd Palino

Print length
485 pages
Publisher
O'Reilly Media
Reading level
Intermediate
ISBN-13
978-1492043089
Published
December 14, 2021
Written for
Backend engineers, data engineers and site reliability engineers building or running Kafka-based streaming systems.

Synopsis

Kafka: The Definitive Guide comes from engineers at Confluent and LinkedIn who worked on Apache Kafka, the distributed event-streaming platform that moves logs, metrics, user activity and transactions between systems in real time. The 2017 first edition, by Kafka co-creator Neha Narkhede with Gwen Shapira and Todd Palino, explains publish-subscribe messaging, then walks through installing brokers, writing producers and consumers, and the internals behind Kafka's guarantees: replication, request handling, the controller and the storage layer. Chapters on reliable delivery, data pipelines with Kafka Connect, cross-cluster mirroring, administration, monitoring and stream processing with Kafka Streams complete the book.

The authors favor explanation over recipes, showing how configuration choices affect durability, ordering and throughput so readers can reason about trade-offs instead of copying settings. A 2021 second edition, by Shapira, Palino, Rajini Sivaram and Krit Petty, added chapters on the AdminClient API, transactions, newer security features and tooling changes. For AI and machine learning teams, Kafka often carries the real-time events that feed feature pipelines and online models, so the book doubles as an introduction to streaming data infrastructure. Neither edition reflects Kafka 4.0, released in March 2025, which removed ZooKeeper entirely in favor of KRaft metadata management.

Scorecard

Editorial score

How it scored

Editorial score: 74.5 / 1005 weighted criteria

  • Influence30%

    22.5/30
  • Authority25%

    22.5/25
  • Practical value20%

    15.0/20
  • Currency15%

    7.5/15
  • Readability10%

    7.0/10
  • Editorial score

    74.5/100

How the rank was computed

Rank No. 305 of 500 comes from a composite of 52.9: observable signals 19.3 of 55 points plus editorial review 33.6 of 45 points.

All 11 inputs
InputPoints
Observable signals19.3/55
Reach (Amazon ratings volume)9.1/16.5
Consensus across best-of lists2.8/13.8
Awards and bestseller evidence0.0/8.3
Reader rating, sample-adjusted6.6/8.3
Amazon sales rank0.8/5.5
Recency0.0/2.8
Editorial review33.6/45
Author authority10.1/11.3
Influence10.1/13.5
Practical value6.8/9.0
Readability3.2/4.5
Currency3.4/6.8

Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.

Verdict

A thorough practitioner guide to Apache Kafka by engineers who helped build it, strong on internals and reliability but predating ZooKeeper's removal.

Strengths

  • Written by engineers who built Kafka
  • Deep explanation of internals and reliability
  • Covers both development and production operations

Keep in mind

Both editions predate Kafka 4.0's removal of ZooKeeper, so cluster setup and administration chapters need supplementing with current documentation.

Key takeaways

  1. Kafka's durability and ordering guarantees depend on configuration choices, such as replication factor, acknowledgments and in-sync replicas, that teams must set deliberately.
  2. Producers and consumers each carry responsibility for reliable delivery; consumer offset commits determine whether messages are processed once, twice or not at all.
  3. Kafka Connect and MirrorMaker let organizations move data between systems and clusters without writing custom integration code for every source and destination.
  4. Monitoring broker, producer and consumer metrics, including consumer lag, is essential to running Kafka clusters reliably in production.

Notable facts

  • Confluent distributes Kafka: The Definitive Guide as a free e-book download. Source (opens in a new tab)
  • I Programmer reviewer Kay Ewbank rated the first edition 4.5 out of 5 and called it a very good book for anyone who needs to learn Kafka. Source (opens in a new tab)

About the author

Neha Narkhede, Gwen Shapira and Todd Palino

Neha Narkhede co-created Apache Kafka while leading streams infrastructure at LinkedIn, then co-founded Confluent in 2014, serving as CTO and later chief product officer, and in 2021 co-founded the risk-detection company Oscilar, where she is CEO. MIT Technology Review named her an Innovator Under 35. Co-authors are Gwen Shapira, a Kafka committer, and Todd Palino, a LinkedIn site reliability engineer who created the Burrow monitoring tool.

Questions

What is Kafka: The Definitive Guide about?
It explains Apache Kafka, the distributed event-streaming platform, for developers and operators. Topics include publish-subscribe messaging, producers and consumers, Kafka's replication and storage internals, reliable delivery, data pipelines with Kafka Connect, cross-cluster mirroring, administration, monitoring and stream processing with Kafka Streams. The authors focus on why Kafka behaves as it does, so readers can make sound configuration and design decisions.
Should I read the first or second edition of Kafka: The Definitive Guide?
Choose the 2021 second edition if you can. It adds chapters on the AdminClient API, transactions, newer security features and tooling changes. Neither edition covers Kafka 4.0, released in March 2025, which removed ZooKeeper and runs only in KRaft mode, so pair either one with the current Apache Kafka documentation for cluster setup.
Is Kafka: The Definitive Guide relevant to AI and machine learning teams?
Yes, indirectly. Many organizations use Kafka to stream events, logs and transactions into feature pipelines, real-time inference services and analytics systems. The book does not teach machine learning, but it explains the delivery guarantees, ordering and scaling behavior that determine whether streaming data reaches models reliably. ML engineers building real-time systems will find it a solid infrastructure reference.

Sources

As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.