No. 332 of 500 No. 45 in Hands-On AI & Programming

Spark

Big Data Processing Made Simple

By Bill Chambers and Matei Zaharia

Published
April 3, 2018
Print length
603 pages
Reading level
Intermediate
ISBN-13
978-1491912218
Publisher
O'Reilly Media
Written for
Data engineers and ML engineers who build or operate large-scale Spark pipelines.

Synopsis

Spark: The Definitive Guide, by Databricks product manager Bill Chambers and Apache Spark creator Matei Zaharia, is a comprehensive manual for the open-source engine used for large-scale data processing, analytics and machine learning pipelines. It opens with a gentle overview of big data and Spark, then concentrates on the structured APIs (DataFrames, Datasets and Spark SQL) through worked examples in Python, Scala and SQL. Later sections descend into the low-level RDD API, explain how Spark executes work on a cluster, and cover deploying, monitoring, debugging and tuning applications in production.

Further parts treat Structured Streaming for end-to-end streaming applications and MLlib for preprocessing, classification, regression, recommendation and other machine learning tasks, along with graph analytics and the wider Spark ecosystem. Chambers wrote the book in Databricks notebooks, drafting roughly a chapter a week, and the result works as both a tutorial and a reference. Readers gain a working model of how Spark plans and distributes computation, which helps when diagnosing slow jobs on any version. The book targets Spark 2.x, however, and later releases changed defaults, added Spark Connect and made ANSI SQL mode standard, so readers should check APIs against current documentation.

Scorecard

Editorial score

How it scored

Editorial score: 72.3 / 1005 weighted criteria

  • Influence30%

    22.5/30
  • Authority25%

    22.5/25
  • Practical value20%

    14.0/20
  • Currency15%

    6.8/15
  • Readability10%

    6.5/10
  • Editorial score

    72.3/100

How the rank was computed

Rank No. 332 of 500 comes from a composite of 51.8: observable signals 19.4 of 55 points plus editorial review 32.4 of 45 points.

All 11 inputs
InputPoints
Observable signals19.4/55
Reach (Amazon ratings volume)9.8/16.5
Consensus across best-of lists2.8/13.8
Awards and bestseller evidence0.0/8.3
Reader rating, sample-adjusted6.1/8.3
Amazon sales rank0.7/5.5
Recency0.0/2.8
Editorial review32.4/45
Author authority10.1/11.3
Influence10.1/13.5
Practical value6.3/9.0
Readability2.9/4.5
Currency3.0/6.8

Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.

Verdict

A thorough Spark 2 reference co-written by the framework's creator, strong on structured APIs and operations but predating Spark 3.0 and 4.0.

Strengths

  • Written by Spark's creator and Databricks staff
  • Covers APIs, internals, streaming and MLlib
  • Practical guidance on tuning and debugging clusters

Keep in mind

Built around Spark 2.x, it predates Spark 3 and 4 changes such as Spark Connect, and its 600-page breadth can overwhelm casual users.

Key takeaways

  1. Prefer Spark's structured APIs, DataFrames and SQL, over low-level RDDs, because the optimizer can plan and execute them far more efficiently.
  2. Understanding how Spark divides work into jobs, stages and tasks across a cluster is essential for diagnosing slow or failing applications.
  3. Structured Streaming lets developers express streaming computations with the same DataFrame operations used for batch processing, simplifying end-to-end pipelines.
  4. MLlib pipelines combine feature preprocessing, model training and evaluation, letting teams scale classical machine learning across large distributed datasets.

Notable facts

  • Co-author Matei Zaharia's dissertation research, which produced Spark, earned the 2014 ACM Doctoral Dissertation Award. Source (opens in a new tab)
  • Bill Chambers drafted about one chapter a week from November 2016 to mid-2017, and the book went to print on February 6, 2018. Source (opens in a new tab)

About the author

Bill Chambers and Matei Zaharia

Bill Chambers wrote the book while a product manager at Databricks, drafting it in Databricks notebooks. Co-author Matei Zaharia started Apache Spark during his PhD at UC Berkeley, where he is now an associate professor of EECS, and is cofounder and CTO of Databricks; his dissertation won the 2014 ACM Doctoral Dissertation Award. He has also worked on Delta Lake and MLflow.

Questions

What does Spark: The Definitive Guide cover?
It covers Apache Spark from first principles to production: the structured APIs (DataFrames, Datasets and SQL), the low-level RDD API, how Spark runs on a cluster, deployment, monitoring, debugging and tuning, Structured Streaming, and machine learning with MLlib. Examples appear in Python, Scala and SQL, and the book is organized so readers can study the sections relevant to their role.
Is Spark: The Definitive Guide still useful in 2026?
Partly. Its explanations of Spark's execution model, structured APIs and tuning remain sound, because those foundations have carried forward. It was written for Spark 2.x, however, and Spark 4.0 changed important behavior, including enabling ANSI SQL mode by default and expanding Spark Connect. Use it for concepts and architecture, and confirm syntax and configuration against current Apache Spark documentation.
Who should read Spark: The Definitive Guide?
Data engineers, platform engineers and data scientists who run Spark jobs at scale will benefit most, especially those responsible for performance and reliability. It suits readers comfortable with Python or Scala and basic SQL. Newcomers who only need to run occasional queries may prefer a shorter introduction, while teams on Spark 3 or 4 should pair it with newer material.

Sources

As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.