Information Theory, Inference, and Learning Algorithms
Synopsis
David MacKay's textbook rests on a conviction that information theory and machine learning are two faces of one subject, since both concern reasoning under uncertainty and representing data efficiently. The book begins with probability, entropy and Shannon's theorems, then develops data compression, including Huffman and arithmetic coding, and noisy-channel coding before branching into broader topics in information theory. The large middle section on probabilities and inference builds a Bayesian toolkit, covering maximum likelihood, clustering, Laplace's method, Monte Carlo methods, variational approximations, message passing and model comparison through Occam's razor.
A section on neural networks treats single neurons, Hopfield networks, Boltzmann machines and Gaussian processes as inference machines, and the closing chapters explain sparse-graph codes such as low-density parity-check, turbo and digital fountain codes. More than 400 exercises, some with detailed solutions, and digressions into crosswords, codebreaking and the evolutionary value of sex keep the tone conversational. The neural network material predates deep learning, so readers will not find convolutional networks or transformers here. What they gain is a durable probabilistic way of thinking that underlies much of modern generative modeling, from variational autoencoders to the view of language modeling as compression. Cambridge University Press still sells the print edition, and MacKay's site hosts the full text for on-screen reading.
Scorecard
Editorial score
How it scored
Editorial score: 79.8 / 1005 weighted criteria
- 25.5/30
Influence30%
- 23.8/25
Authority25%
- 13.0/20
Practical value20%
- 10.5/15
Currency15%
- 7.0/10
Readability10%
- 79.8/100
Editorial score
How the rank was computed
Rank No. 208 of 500 comes from a composite of 56.4: observable signals 20.4 of 55 points plus editorial review 36.0 of 45 points.
All 11 inputs
| Input | Points |
|---|---|
| Observable signals | 20.4/55 |
| Reach (Amazon ratings volume) | 8.4/16.5 |
| Consensus across best-of lists | 5.5/13.8 |
| Awards and bestseller evidence | 0.0/8.3 |
| Reader rating, sample-adjusted | 5.4/8.3 |
| Amazon sales rank | 1.1/5.5 |
| Recency | 0.0/2.8 |
| Editorial review | 36.0/45 |
| Author authority | 10.7/11.3 |
| Influence | 11.5/13.5 |
| Practical value | 5.9/9.0 |
| Readability | 3.2/4.5 |
| Currency | 4.7/6.8 |
Each criterion is scored from 1 to 10 and weighted, and the points add up to the editorial score. The rank combines observable signals (55 percent) with this score (45 percent). Read the methodology.
Verdict
A lucid, freely readable textbook that unites information theory, Bayesian inference and neural networks; demanding, but rewarding for mathematically prepared readers.
Strengths
- Unifies information theory with Bayesian machine learning
- More than 400 exercises for self-study
- Full text free to read online
Keep in mind
It demands comfort with probability and calculus, and its neural network chapters predate deep learning, so readers seeking modern architectures must look elsewhere.
Key takeaways
- Information theory and machine learning share foundations, because compressing data well and predicting it well are closely linked problems.
- Bayesian inference offers one framework for learning, model comparison and uncertainty, with Occam's razor emerging naturally from probability theory rather than as an added rule.
- When exact inference is intractable, Monte Carlo methods, variational approximations and message passing provide practical approximate alternatives.
- Sparse-graph codes such as low-density parity-check and turbo codes approach Shannon's limit and show inference algorithms at work in communications engineering.
Notable facts
- Caltech's Bob McEliece called it 'an instant classic,' and Zoubin Ghahramani and Peter Dayan of UCL's Gatsby Unit called it 'an extraordinary and important book.' Source (opens in a new tab)
- MacKay kept the complete book available for on-screen reading on his website after Cambridge University Press published it in September 2003. Source (opens in a new tab)
About the author
Questions
Is Information Theory, Inference, and Learning Algorithms free?
Is MacKay's book still relevant for machine learning in 2026?
Who should read Information Theory, Inference, and Learning Algorithms?
Related books
- Probabilistic Robotics Sebastian Thrun, Wolfram Burgard and Dieter Fox
- Data-Driven Science and Engineering Steven L. Brunton and J. Nathan Kutz
- Probabilistic Machine Learning Kevin P. Murphy
- Elements of Causal Inference Jonas Peters, Dominik Janzing and Bernhard Schölkopf
- Practical Statistics for Data Scientists Peter Bruce, Andrew Bruce and Peter Gedeck
- High-Dimensional Probability Roman Vershynin
Sources
As an Amazon Associate, Best AI Books earns from qualifying purchases. Best AI Books is sponsored by Iternal.ai, whose founder and CEO, John Byron Hanby IV, wrote The AI Strategy Blueprint, our #1 ranked book. How we rank books.






