电子书InformationTheoryInferenceandLearningAlgorithms

5星 · 超过95%的资源需积分: 9 33 浏览量更新于2023-03-03 评论 4 收藏 11.28MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v 1 Introduction to Information Theory . . . . . . . . . . . . . 3 2 Probability, Entropy, and Inference . . . . . . . . . . . . . . 22 3 More about Inference . . . . . . . . . . . . . . . . . . . . . 48 I Data Compression . . . . . . . . . . . . . . . . . . . . . . 65 4 The Source Coding Theorem . . . . . . . . . . . . . . . . . 67 5 Symbol Codes . . . . . . . . . . . . . . . . . . . . . . . . . 91 6 Stream Codes . . . . . . . . . . . . . . . . . . . . . . . . . . 110 7 Codes for Integers . . . . . . . . . . . . . . . . . . . . . . . 132 II Noisy-Channel Coding . . . . . . . . . . . . . . . . . . . . 137 8 Dependent Random Variables . . . . . . . . . . . . . . . . . 138 9 Communication over a Noisy Channel . . . . . . . . . . . . 146 10 The Noisy-Channel Coding Theorem . . . . . . . . . . . . . 162 11 Error-Correcting Codes and Real Channels . . . . . . . . . 177 III Further Topics in Information Theory . . . . . . . . . . . . . 191 12 Hash Codes: Codes for Ecient Information Retrieval . . 193 13 Binary Codes . . . . . . . . . . . . . . . . . . . . . . . . . 206 14 Very Good Linear Codes Exist . . . . . . . . . . . . . . . . 229 15 Further Exercises on Information Theory . . . . . . . . . . 233 16 Message Passing . . . . . . . . . . . . . . . . . . . . . . . . 241 17 Communication over Constrained Noiseless Channels . . . 248 18 Crosswords and Codebreaking . . . . . . . . . . . . . . . . 260 19 Why have Sex? Information Acquisition and Evolution . . 269 IV Probabilities and Inference . . . . . . . . . . . . . . . . . . 281 20 An Example Inference Task: Clustering . . . . . . . . . . . 284 21 Exact Inference by Complete Enumeration . . . . . . . . . 293 22 Maximum Likelihood and Clustering . . . . . . . . . . . . . 300 23 Useful Probability Distributions . . . . . . . . . . . . . . . 311 24 Exact Marginalization . . . . . . . . . . . . . . . . . . . . . 319 25 Exact Marginalization in Trellises . . . . . . . . . . . . . . 324 26 Exact Marginalization in Graphs . . . . . . . . . . . . . . . 334 27 Laplace's Method . . . . . . . . . . . . . . . . . . . . . . . 341 28 Model Comparison and Occam's Razor . . . . . . . . . . . 343 29 Monte Carlo Methods . . . . . . . . . . . . . . . . . . . . . 357 30 Ecient Monte Carlo Methods . . . . . . . . . . . . . . . . 387 31 Ising Models . . . . . . . . . . . . . . . . . . . . . . . . . . 400 32 Exact Monte Carlo Sampling . . . . . . . . . . . . . . . . . 413 33 Variational Methods . . . . . . . . . . . . . . . . . . . . . . 422 34 Independent Component Analysis and Latent Variable Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437 35 Random Inference Topics . . . . . . . . . . . . . . . . . . . 445 36 Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . 451 37 Bayesian Inference and Sampling Theory . . . . . . . . . . 457 V Neural networks . . . . . . . . . . . . . . . . . . . . . . . . 467 38 Introduction to Neural Networks . . . . . . . . . . . . . . . 468 39 The Single Neuron as a Classier . . . . . . . . . . . . . . . 471 40 Capacity of a Single Neuron . . . . . . . . . . . . . . . . . . 483 41 Learning as Inference . . . . . . . . . . . . . . . . . . . . . 492 42 Hopeld Networks . . . . . . . . . . . . . . . . . . . . . . . 505 43 Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . 522 44 Supervised Learning in Multilayer Networks . . . . . . . . . 527 45 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . 535 46 Deconvolution . . . . . . . . . . . . . . . . . . . . . . . . . 549 VI Sparse Graph Codes . . . . . . . . . . . . . . . . . . . . . 555 47 Low-Density Parity-Check Codes . . . . . . . . . . . . . . 557 48 Convolutional Codes and Turbo Codes . . . . . . . . . . . . 574 49 Repeat{Accumulate Codes . . . . . . . . . . . . . . . . . . 582 50 Digital Fountain Codes . . . . . . . . . . . . . . . . . . . . 589 VII Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . 597 A Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598 B Some Physics . . . . . . . . . . . . . . . . . . . . . . . . . . 601 C Some Mathematics . . . . . . . . . . . . . . . . . . . . . . . 605 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620

资源详情

资源评论

资源推荐

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981

You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.

Information Theory, Inference, and Learning Algorithms

David J.C. MacKay

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981

You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

1 Introduction to Information Theory . . . . . . . . . . . . . 3

2 Probability, Entropy, and Inference . . . . . . . . . . . . . . 22

3 More about Inference . . . . . . . . . . . . . . . . . . . . . 48

I Data Compression . . . . . . . . . . . . . . . . . . . . . . 65

4 The Source Coding Theorem . . . . . . . . . . . . . . . . . 67

5 Symbol Codes . . . . . . . . . . . . . . . . . . . . . . . . . 91

6 Stream Codes . . . . . . . . . . . . . . . . . . . . . . . . . . 110

7 Codes for Integers . . . . . . . . . . . . . . . . . . . . . . . 132

II Noisy-Channel Coding . . . . . . . . . . . . . . . . . . . . 137

8 Dependent Random Variables . . . . . . . . . . . . . . . . . 138

9 Communication over a Noisy Channel . . . . . . . . . . . . 146

10 The Noisy-Channel Coding Theorem . . . . . . . . . . . . . 162

11 Error-Correcting Codes and Real Channels . . . . . . . . . 177

III Further Topics in Information Theory . . . . . . . . . . . . . 191

12 Hash Codes: Codes for Eﬃcient Information Retrieval . . 193

13 Binary Codes . . . . . . . . . . . . . . . . . . . . . . . . . 206

14 Very Good Linear Codes Exist . . . . . . . . . . . . . . . . 229

15 Further Exercises on Information Theory . . . . . . . . . . 233

16 Message Passing . . . . . . . . . . . . . . . . . . . . . . . . 241

17 Communication over Constrained Noiseless Channels . . . 248

18 Crosswords and Codebreaking . . . . . . . . . . . . . . . . 260

19 Why have Sex? Information Acquisition and Evolution . . 269

IV Probabilities and Inference . . . . . . . . . . . . . . . . . . 281

20 An Example Inference Task: Clustering . . . . . . . . . . . 284

21 Exact Inference by Complete Enumeration . . . . . . . . . 293

22 Maximum Likelihood and Clustering . . . . . . . . . . . . . 300

23 Useful Probability Distributions . . . . . . . . . . . . . . . 311

24 Exact Marginalization . . . . . . . . . . . . . . . . . . . . . 319

25 Exact Marginalization in Trellises . . . . . . . . . . . . . . 324

26 Exact Marginalization in Graphs . . . . . . . . . . . . . . . 334

27 Laplace’s Method . . . . . . . . . . . . . . . . . . . . . . . 341

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981

You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.

28 Model Comparison and Occam’s Razor . . . . . . . . . . . 343

29 Monte Carlo Methods . . . . . . . . . . . . . . . . . . . . . 357

30 Eﬃcient Monte Carlo Methods . . . . . . . . . . . . . . . . 387

31 Ising Models . . . . . . . . . . . . . . . . . . . . . . . . . . 400

32 Exact Monte Carlo Sampling . . . . . . . . . . . . . . . . . 413

33 Variational Methods . . . . . . . . . . . . . . . . . . . . . . 422

34 Independent Component Analysis and Latent Variable Mod-

elling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437

35 Random Inference Topics . . . . . . . . . . . . . . . . . . . 445

36 Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . 451

37 Bayesian Inference and Sampling Theory . . . . . . . . . . 457

V Neural networks . . . . . . . . . . . . . . . . . . . . . . . . 467

38 Introduction to Neural Networks . . . . . . . . . . . . . . . 468

39 The Single Neuron as a Classiﬁer . . . . . . . . . . . . . . . 471

40 Capacity of a Single Neuron . . . . . . . . . . . . . . . . . . 483

41 Learning as Inference . . . . . . . . . . . . . . . . . . . . . 492

42 Hopﬁeld Networks . . . . . . . . . . . . . . . . . . . . . . . 505

43 Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . 522

44 Supervised Learning in Multilayer Networks . . . . . . . . . 527

45 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . 535

46 Deconvolution . . . . . . . . . . . . . . . . . . . . . . . . . 549

VI Sparse Graph Codes . . . . . . . . . . . . . . . . . . . . . 555

47 Low-Density Parity-Check Codes . . . . . . . . . . . . . . 557

48 Convolutional Codes and Turbo Codes . . . . . . . . . . . . 574

49 Repeat–Accumulate Codes . . . . . . . . . . . . . . . . . . 582

50 Digital Fountain Codes . . . . . . . . . . . . . . . . . . . . 589

VII Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . 597

A Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 598

B Some Physics . . . . . . . . . . . . . . . . . . . . . . . . . . 601

C Some Mathematics . . . . . . . . . . . . . . . . . . . . . . . 605

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981

You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.

Preface

This book is aimed at senior undergraduates and graduate students in Engi-

neering, Science, Mathematics, and Computing. It expects familiarity with

calculus, probability theory, and linear algebra as taught in a ﬁrst- or second-

year undergraduate course on mathematics for scientists and engineers.

Conventional courses on information theory cover not only the beauti-

ful theoretical ideas of Shannon, but also practical solutions to communica-

tion problems. This book goes further, bringing in Bayesian data modelling,

Monte Carlo methods, variational methods, clustering algorithms, and neural

networks.

Why unify information theory and machine learning? Because they are

two sides of the same coin. In the 1960s, a single ﬁeld, cybernetics, was

populated by information theorists, computer scientists, and neuroscientists,

all studying common problems. Information theory and machine learning still

belong together. Brains are the ultimate compression and communication

systems. And the state-of-the-art algorithms for both data compression and

error-correcting codes use the same tools as machine learning.

How to use this book

The essential dependencies between chapters are indicated in the ﬁgure on the

next page. An arrow from one chapter to another indicates that the second

chapter requires some of the ﬁrst.

Within Parts I, II, IV, and V of this book, chapters on advanced or optional

topics are towards the end. All chapters of Part III are optional on a ﬁrst

reading, except perhaps for Chapter 16 (Message Passing).

The same system sometimes applies within a chapter: the ﬁnal sections of-

ten deal with advanced topics that can be skipped on a ﬁrst reading. For exam-

ple in two key chapters – Chapter 4 (The Source Coding Theorem) and Chap-

ter 10 (The Noisy-Channel Coding Theorem) – the ﬁrst-time reader should

detour at section 4.5 and section 10.4 respectively.

Pages vii–x show a few ways to use this book. First, I give the roadmap for

a course that I teach in Cambridge: ‘Information theory, pattern recognition,

and neural networks’. The book is also intended as a textbook for traditional

courses in information theory. The second roadmap shows the chapters for an

introductory information theory course and the third for a course aimed at an

understanding of state-of-the-art error-correcting codes. The fourth roadmap

shows how to use the text in a conventional course on machine learning.

剩余639页未读，继续阅读

zld_225

2018-03-23

好资源，学习ml的经典书

阿里茫茫

粉丝: 1
资源: 2

会员权益专享

电子书Information Theory Inference and Learning Algorithms

评论17

会员权益专享

最新资源

电子书Information Theory Inference and Learning Algorithms

评论17

Information.Theory.Inference.and.Learning.Algorithms.pdf

信息论与编码理论 王育民主编 第二章答案

Information Theory, Inference, and Learning Algorithms

关于信息论的五个参考文献

mackay构造校验矩阵

CPU THREADING AND TORCHSCRIPT INFERENCE介绍

Deep Learning Toolbox

linear statistical inference and its applications pdf

integer quantization for deep learning inference: principles and empirical e

computer vision: models, learning, and inference 配套课程

causal inference for statistics social and biomedical sciences 下载

def inference(self, image):

我想学习计算机视觉 帮我推荐几本书籍

inference in Bayes

jetson-inference安装

import torch

All of Statistics

from sb3_contrib import MaskablePPO

会员权益专享

最新资源

信息论与编码理论王育民主编第二章答案

我想学习计算机视觉帮我推荐几本书籍